A composable autoencoder-based algorithm for accelerating numerical simulations
Authors: Rishikesh Ranade
Derek Christopher Hill
Haiyang He
Amir Maleki
Norman Chang
Jay Pathak
Venue: ICLR 2022 Submitted (2022)
Keywords: numerical simulations
machine learning

Abstract: Numerical simulations for engineering applications solve partial differential equations (PDE) to model various physical processes. Traditional PDE solvers are very accurate but computationally costly. On the other hand, Machine Learning (ML) methods offer a significant computational speedup but face challenges with accuracy and generalization to different PDE conditions, such as geometry, boundary conditions, initial conditions and PDE source terms. In this work, we propose a novel ML-based approach, CoAE-MLSim, which is an unsupervised, lower-dimensional, local method, that is motivated from key ideas used in commercial PDE solvers. This allows our approach to learn better with relatively fewer samples of PDE solutions. The proposed ML-approach is compared against commercial solvers for better benchmarks as well as latest ML-approaches for solving PDEs. It is tested for a variety of complex engineering cases to demonstrate its computational speed, accuracy, scalability, and generalization across different PDE conditions. The results show that our approach captures physics accurately across all metrics of comparison (including measures such as results on section cuts and lines).

Under review as a conference paper at ICLR 2022
A COMPOSABLE AUTOENCODER-BASEDI ALGORITHM FOR ACCELERATING NUMERICAL SIMULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Numerical simulations for engineering applications solve partial differential equa- tions (PDE) to model various physical processes. Traditional PDE solvers are very accurate but computationally costly. On the other hand, Machine Learn- ing (MI.) methods offer a significant computational speedup but face challenges with accuracy and generalization to different PDE conditions, such geometry. boundary conditions, initial conditions and PDE source terms. this work, we propose novel ML-based approach, CoAE-MLSim (Composable AutoEncoder Machine Learning Simulation), which an unsupervised, dewer-dimensional, lo- cal method, that is motivated from key ideas used in commercial PDE solvers. This allows our approach to learn better with relatively fewer samples of PDE solutions. The proposed ML-approach is compared against commercial solvers for better benchmarks well as latest ML-approaches for solving PDEs. It is tested for variety of complex engincering cases demonstrate its computational speed, accuracy, scalability, and generalization across different PDE conditions. The results show that our approach captures physics accurately across all metrics of comparison (including measures such as results on section cuts and lines).
INTRODUCTION
Numerical solutions to partial differential equations (PDEs) are dependent on PDE conditions such as, geometry of the computational domain, boundary conditions, initial conditions and source terms. Commercial PDE solvers have shown tremendous success in accurately modeling PDEs for wide range of applications. These solvers generalize across different PDE conditions but can be computationally slow. Moreover, their solutions are not reusable and need to be solved from scratch every time the PDE conditions are changed.
The idea of using Machine Learning (ML) with PDEs has been explored for several decades (Crutch- heeld & McNamara 1987 Kevrekidis et al. 2003) but with recent developments in computing hard- and ML techniques, these efforts have grown immensely. Although ML approaches are compu- tationally fast, they fall short of traditional PDE solvers with respect to accuracy and generalization wide range of PDE conditions. Most data-driven and physics- chysicc-conttrained approaches employ static-inferencing strategies, where a mapping function is learnt between PDE solutions and corre- sponding conditions. In many cases, PDE conditions are sparse and high-dimensional,, and hence, difficult to generalize. Additionally, current ML approaches do not make use of the key ideas from traditional PDE solvers such as, domain decomposition, solver methods, numerical discretization. constraint equations, symmetry evaluations and tighter non statistical evaluation metrics, which were established over several decades of research and development. this work, we propose novel ML approach that is motivated from such ideas and relies on dynamic inferencing strategies that afford the possibility of seamless coupling with traditional PDE solvers, when necessary.
The proposed ML-approach, CoAE-MLSim (Composable AutoEncoder Machine Learning Simulation), operates at the level of local subdomains, which consist a group of pixels in 2D or voxels in 3D (for example 8 or 16 in each spatial direction). As shown in Figure [] the CoAE-MLSim has main components: A) lear solutions on local subdomains, B) learn the rules of how a group of local subdomains connect together yield
Figure 1: CoAE-MLSim components

---

Under review as a conference paper at ICLR 2022
locally consistent solutions & C) deploy an it- erative algorithm to establish local consistency across all groups of subdomains in the entire computational domain. The solutions subdomains are learnt into low-dimensional representations using solution autoencoders, while the rules between groups of subdomains, corresponding flux conservation between PDE solution is learnt using flux conservation autoencoders. The autoencoders are trained a priori on local subdomains using very few PDE solution samples. During inference, the pretrained autoencoders are combined with itera- tive solution algorithms to determine PDE solutions for different PDEs over wide range of sparse, high-dimensional PDE conditions. The solution strategy the CoAE-MLSing approach very sim- ilar to traditional PDE solvers, moreover the iterative inferencing strategy allows for coupling with traditional PDE solvers to accelerate convergence and improve accuracy and generalizability.
Significant contributions of this work:
1. The CoAE-MLSim approach combines traditional PDE solver strategies with ML tech- niques accurately model numerical simulations.
2. Our approach operates local subdomains and solves PDEs in low-dimensional space. This enables generalization to arbitrary PDE conditions within a high-dimensional distri- bution.
3. The CoAE-ML approach training corresponds to training the various autoencoders local subdomains and hence, it contains very few parameters and requires less data.
4. The iterative inferencing algorithm is completely unsupervised and allows for coupling with traditional PDE solvers.
5. Finally, we show generalization of our model for wide variations in sparse, high dimen- sional PDE conditions using evaluation metrics in tune with commercial solvers.
2 RELATED WORKS
The of ML for solving PDEs has gained tremendous traction in the past few years. Most of the literature has revolved around improving neural network architectures and optimization techniques to enable generalizable and accurate learning of PDE solutions. More recently, there has been lot of focus on learning mesh independent, infinite dimensional operators with neural networks (NNs)
Bhattacharya al. (2020); Anandkumar al. [2020)]
: Li al. (2020b): Patel al. 2021 : Lu al.
(2021b): Li et al. (2020a). The neural operators are trained from high-fidelity solutions generated by traditional PDE solvers on mesh of specific resolution and do not require any knowledge of
the PDE. From the perspective of mesh independent learning, Battaglia et al. (2018) introduced
graph network architecture, which has proven be effective in solving dynamical systems directly
on unstructured computational meshes. (Sanalez-Gonzalez et al. (2020) and Pfaff et al. (2020) use the graph network for robustly solving transient dynamics on arbitrary meshes and to accurately capture the transient solution trajectories. For dynamical systems, the neural operators and mesh based learning strategies have shown reasonable prediction capabilities, however, the generalizabil- ity of these methods to different PDE conditions remains to be seen. Another direction of research falls under the category of training neural networks with physics constrained optimization. Research in this space involves constraining neural networks with additional physics- based losses introduced through loss re-engineering. PDE constrained optimization have shown to improve interpretability and accuracy of results. RRaissi et al. (2019) and Raissi & Karniadakis (2018) introduced the frame- work of physics-informed neural network (PINN) to constrain neural networks with PDE derivatives computed using Automatic Differentiation (AD) (Baydin al. 2018 the past couple of years, the PINN framework has been extended to solve complicated PDEs representing complex physics
[Jin et al.
2021 Mao et al., 2020 Rao et al.
2020 Wu et al. 2018 Qian et al. 2020 Dwivedi al.
2021 Haghighat et al. 2021 Haghighat & Juanes
2021 Nabian et al. 2021 Kharazmi et al. 2021 Cai et al. 202 by Bode et al. 202 Taghizadeh et al. 2021 et al. 202 Shukla et al. 2021
Hennigh et al. 2020 Li et al.
2021 More recently, alternate approaches that use discretization
techniques using higher order derivatives and specialize numerical schemes to compute derivatives
have shown to provide better regularization for faster convergence (Ranade et al. 2021b Gao al.)
2021 Wandel et al. |020 He & Pathak 2020). However, the use of optimization techniques to solve
PDES, although accurate, has proved to be extremely slow as compared to traditional solvers. There
2

---

Under review as a conference paper at ICLR 2022
is a large body of work related to training neural networks in tandem with differentiable PDE solvers in the loop to improve long range time predictions. 'hee use of differentiable solvers to provide ac- curate estimates of adjoints has shown to improve learning and provide better control of PDE
solutions and transient system dynamics (Amos & Kolter 2017 Um et al. 2020; de Avila Belbute-
Peres al. 2018: Toussaint et al.) 20188 Wang et al. 2020 Holl al. 2020 Portwood al. 2019)- These frameworks are useful in providing rapid feedback to neural networks to improve conver-
gence stability and enable efficient exploration of solution space, Bar--Sinii al. 2019). Xue et al.
(020) and Kochkov et al. (202T) train neural networks in conjunction with a differentiable solver to learn high-fidllig PDE discretization on coarse grids. [Singh al] (2017) and (Hollan et al.) (2019) use differentiable solvers to tune model parameters in a live simulation. However, these methods require a differentiable solver the training loop and commercial solvers that solve industrial scale problems may not be differentiable. All of the methods discussed above try to learn the learn the dynamics of the PDE in the solution space on entire computational domain, which can be a chal- lenging task as domains can be eigh-dimensional with complicated physics. For many problems, local learning from smaller restricted domains have proved to accelerate learning of neural networks and provide better accuracy and generalization. Lu et al. (202 la) and Wang al. (202T) learn lo- calized domains but infer on larger computational domains using a stitching process. Bar-Sinai et al.) (2019) and Kochkov et al. 1 learn coefficients of numerical discretization schemes from high fidelity data, which is sub-sampled on coarse grids. [Beatson et al. (2020) learns surrogate models for smaller components allow for cheaper simulations. Other methods compress PDE solutions on to lower-dimensional manifolds. This has shown to improve accuracy and generalization capa- bility of neural networks (Wiewel et al., [0200] Maulik et al.) 2020 Kim et al. 2019] Murata et al. 2020 Fukami et al.) [0200] Ranade et al., (021a). The ideas proposed in this work share the same R goal with the aforementioned literature to accelerate numerical simulations without compromising on accuracy and generalizability. Our work draws inspiration from techniques such as local and latent space learning to solve PDEs in both transient and steady-state settings. More importantly, the approach proposed here relies heavily on unsupervised techniques such autoencoders to solve PDEs Ranade et al. (201 by: Maleki et al. (2021) and mainly focus on leveraging knowledge from traditional solvers develop robust, stable, accurate and generalizable machine learning system.
3
COAE-MLSIM MODEL DETAILS
3.1 SIMILARITIES WITH TRADITIONAL PDE SOLVERS
Consider a set of coupled PDEs with n solution variables. For the sake of notation simplicity, we take = 2, such that u(x, t) and v(x y, 2, f) are defined on computational domain Q with boundary conditions specified on the boundary of the computational domain, Q, should be noted that extension to more solution variables is trivial. The coupled PDEs are defined as follows:
(u F1 = 0; L2(u, v) F2 = 0
where, L1, Le denote PDE operators and F1, F2 represent PDE source terms. The PDE operators can vary for different PDEs. For example, in a non linear PDE such as the unsteady, incompressible Navier-Stokes equation the operator, L = Đt + a. V - A.4
Traditional PDE solvers solve PDEs given in Eq. I by representing solutions variables, u, v, and their linear and non-linear spatio-temporal derivatives on a discretized computational domain. The numerical approximations on discrete domains are computed on finite number of computational el- ements known as a mesh using techniques such as Finite Difference Method (FDM), Finite Volume Method (FVM) and Finite Element Method (FEM). These solvers use iterative solution algorithms to conserve fluxes between neighboring computational elements and determine consistent PDE so- lutions over the entire domain at convergence. The CoAE-MLSim approach designed to perform similar operations but at the level of subdomains with assistance from ML techniques.
3.2 COAE-MLSIM FOR STEADY-STATE PDES
Steady-state PDEs correspond to partial differential equations without time dependencies. The PDE solutions in these equations are primarily governed by PDE conditions, which can have high- dimensional and sparse representations. The solution algorithm of CoAF-MLSim approach for
3

---

Under review as a conference paper at ICLR 2022
steady-state PDEs is shown in Fig. and Alg. Similar to traditional solvers, the CoAE-MLSim approach discretizes the computational domain into several subdomains. Each subdomain has a constant physical size and represents both PDE solutions and conditions. For example, subdomain cutring across the cylinder in Fig. 2 represents the geometry and boundary conditions of the part- cylinder. In steady-state problem, the computational domain is initialized with initial PDE solu- tions that are either randomly sampled or generated by coarse-grid PDE solvers. The initial solution used to encode
shown with randomly oriented flow vectors in Fig. 2 Pre-trained encoders the initial solutions well as user-specified PDE conditions into Lower-dimensions latent vectors, Nu 77 corresponding to PDE solutions and g. b, s corresponding to geometry, boundary conditions and sourceterms, respectively. The flux conservation is iteratively established by passing batches of neighboring subdomain latent vectors through a pre-trained flux conservation autoencoder (+). In each iteration, the algorithm loops over all the subdomains, gathers neighbors for a given subdomain and concatenates the neighboring latent vectors corresponding to PDE solutions and conditions. The concatenated latent vectors evaluated through the pre-trained flux conservation autoencoder to obtain the new solution latent vector (7a 7//e) state on each subdomain. Subdomains on the bound- ary have fewer neighbors and latent vectors zero padded in such cases. The flux conservation autoencoder couples all the solution variables with PDE conditions to ensure that all the dependen- cies captured. The iteration stops when the L2 norm of change in solution latent vectors meets a specified tolerance, otherwise the latent vectors are updated and the iteration continues. The en- codings of the PDE conditions are not updated and help in steering the solution latent vectors to an equilibrium state that is decoded to PDE solutions using pre-trained decoders (gw. on the computa- tional domain. The converged solution Fig. 2 is represented with flow vectors that are consistent with neighboring subdomains. The iterative procedure used in the CoAE-MLSim approach can be implemented using several linear or non-linear equation solvers, such as Fixed point iterations (Bai al. 2019), Gauss Siedel, Newton's method etc., that are used in commercial PDE solvers. Physics constrained optimization at inference time might be used to improve convergence robustness and fidelity with physics. The use of autoencoders for compressed solution and condition representation in tandem with iterative inferencing solution algorithm is inspired from Ranade et al. (2021a] b) but the main difference in this work is that the solution procedure is carried on local subdomains as opposed to entire computational domains to align with principles used in traditional PDE solvers.
User Specs.
Each vuldumaai initial PDE solution, geometry, BC and terms
Computational
demain//Di conditions
Fland point Invations
solution
Break
Flux conservation for
domain
sublomains
Encode
T St by
Decode
n: solution encoding, Goometry encoding
boundary condition encoding, 5, socurce escoding
Figure 2: Steady state CoAE-MLSim algorithm
Algorithm 1: Steady state solution methodology of CoAE-MI approach Domain Decomposition: Computational domain 52 + Subdomains Ω Initialize solution all Ω,: ((x)) = 0.0 for all x E Ω
Encode PDE solution and conditions on all Ω2:
1/2 = e. (u(().)), " - (o(o)) mp (b(.))) ns (s(s))
= le
while e > 6 do
for Ω2 E Ω do
Gather neighbors of Z,; And = MasRiesh &right:
nh = 0 nb my m)
Compute L2 norm: e = 1| 7/a Update: 7/ + 7/2 for all Ω2 f
10
Decode PDE solution all Ωg: i gu ())
4

---

Under review as a conference paper at ICLR 2022
3.3 COSE-MLSIM FOR TRANSIENT PDES
The algorithm for transient CoAE-MLSim differs slightly from steady-state. Our approach models transient PDEs using two methods, fully coupled and loosely coupled. The fully coupled analo- gous to the steady-state algorithm where time is considered as an additional dimension in addition to space and modeled using the algorithm described in Alg. [ On the other hand, in the loosely coupled approach the spatial and transient effects
decoupled. The decoupling allows for better modeling of long range time dynamics and results in improved stability and generalizability, This claim has been validated in Appendix A.2.7 The solution methodology shown in Figure corre- sponds to the loosely coupled approach and differs from the steady-state methodology in Figure that it an additional pre-trained time integration network, which integrates the solution latent vectors in time. More details on the time integration network are provided in Section (36 Analogous traditional PDE solvers, a flux conservation is applied after every time integration step. This is very important in establishing local consistency between neighborhood subdomains and minimizing error accumulation resulting from the transient process. Since, flux conservation and time integra- tion networks are trained using the compressed latent vectors of PDE solutions and conditions, the PDE solution and condition autoencoders are required be trained first.
3.4 PDE SOLUTION AND CONDITION AUTOENCODERS
Autoencoders are used to establish conditions
lower-dimensional represented
latent vectors, on local subdomains.
77 Ms. Mg.. for PDE solutions and Figure schematic of the autoen- coder setup used in the CoAF-MI. Mi..
solution Autcensoder
Source term Autoencoder
f
Although the representative subdomains shown are 2-D, same concepts apply higher dimensions. The geometry autoen- coders encode a representation of the ge- ometry into latent vector, 1/g- In this work, we adopt Signed Distance Field (SDF) representation of the geometry because it smooth and differentiable (Maleki et al.) 2021I. Similarly, the PDE source terms are encoded to their respective latent vector, The solution autoencoders are conditioned upon the compressed latent vectors of the PDE conditions by concatenating them with the latent vector (1) of the PDE solutions (Ranade et al. 2021a). We recommend the training of each solution vari- able using different autoencoder to improve accuracy. The autoencoders are trained using sample PDE solutions generated on entire computational domains and then divided into smaller subdomains training.
Dacido:
Booslr
fi
Mg fl
Autoensoder
Prendaed Exsder
Taunder
Figure 4: Autoencoders PDE solutions and condition
3.5 FLUX CONSERVATION AUTOENCODER
Flux conservation autoencoders are responsible for bulk of the calculations in the solution algorithm. Their main function is to disperse solution information throughout the computa- tional domain by transferring information between neighborhood subdomains and from bound- aries and geometries. Figure shows the schematic the flux conservation autoencoder,
subdomuii geametry,
Cumputationl Pe
Flnal solutien
Flued devatiooss
Break down denaia into widdemaias
conservation
Excode:
TiCL by.S)
subdematt
Ticces g. by.s subdomgis
Wizso of
Decode
0: solation excudit,, Geoncty excoiin bouslry condition excoding sounce ensvding
Figure 3: Transient CoAE-MLSim algorithm

---

Under review as a conference paper at ICLR 2022
which operates the latent vectors of PDE solutions and conditions group of neigh- boring subdomains. The imputs and outputs to this network consist of concatenated latent vectors of all solution variables and PDE conditions group of neighboring subdomains.
uses a deep fully connected neural net- work to learn these relationships. The
Felly
samples generated for autoencoders in
Counntel
Section 3.4 are used for training the flux conservation autoencoders as well.
Docelns
3.6 TIME INTEGRATION NETWORK
Figure Flux conservation autoencoder
Figure shows the schematic of the time integration network, which operates on the latent vectors of PDE solutions and condi- tions on a group of neighboring subdomains. The time integration network uses fully connected networks to transform the solution latent vector of the center subdomain (corresponding 6) in time. The input of the time integration network can stack latent vectors from multiple previous time steps, 4, 1, 2... predict the latent vectors for the next time step, l 1+1. 1. We recommend the training of each solution variable using a different network to improve Securacy.Similas to other autoencoders, the time integration network also learns from randomly generate solution samples. More details on all the autoencoders are in appendix A.1
in Fig.
3.7 WHY AUTOENCODERS?
step ",
At dine step '(=]'
Solutions to classical PDEs such as the Laplace equation can represented by homogeneous solu- tions, such as: Φ(x, v) = + 01 a2y az(x² + where, A do, a1 02 ..., On are constant coefficients that can be used to reconstruct the PDE solution on any local subdomain. A can be considered as compressed encoding of the Laplace solutions. Since, is not possible to explicitly derive such compressed encodings for other high dimensional and non-linear PDEs, the CoAF-MLSing ap- proach relies on autoencoders to compute them. It known that non-linear autoencoders with good compression ratios can learn powerful non-linear generalizations (Goodiellow et al. 2016 Rumel- hart et al. 1985 Bank et al.) 2020). Thus, autoencoders enable efficient latent space computations. Autoencoders also have great denoising abilities, which improve robustness and stability, when used in iterative settings (Ranade et al. 2021a). Finally, they are data-efficient and result a small num- ber of learnable model parameters and much faster training. For these reasons, autoencoders form the fundamental building block of the CoAF-ML approach.
Pretaised
Pully
Maoths-tta DNN
Ku Vin 06.5 -i
Pretraised Docodars
+a) (2ry)+ (2x+))
Figure 6: Time integration network
3.8 ATTRIBUTES OF COAE-MLSIM SOLUTION METHODOLOGG
1. Unsupervised algorithm: Although, the autoencoders trained PDE solutions gen- erated for random PDE conditions, the iterative solution procedure described in Section 3.2 is never explicitly taught the process of computing PDE solutions and discovers solutions with a minimal knowledge about the rules of local consistency. This is remarkably similar to how traditional PDE solvers would operate.
2. Local learning: Since the autoencoders trained on local subdomains, they have fewer trainable parameters and need very less data samples (100 - 1000) to learn the dynamics between PDE solutions and high-dimensional PDE conditions.
3. Latent space representation:
The iterative solution procedure is carried out in a com- pressed latent space to achieve solution speed-ups. Furthermore, compressed representa- tions of sparse, high-dimensional PDE conditions improves generalizability.
4. Coupling with traditional PDE solvers: The iterative inferencing strategy and the solu- tion algorithm is designed similar traditional solver and allows for smooth coupling.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS AND RESULTS
This section demonstrates the CoAE-MLSim approach for two use cases, steady-state conjugate heat transfer and transient vortex decay. Additional applications are presented in the appendix sections, [..2.]] [A.2.4] and [A.2.6] All the experiments have varying levels of complexity across geometries, boundary and initial conditions & source terms imposed on the PDE. The source code and the datasets used in our experiments and analysis can be made available upon acceptance.
corresponding different PDEs and
Data generation and training mechanics The data required to train the several autoencoders in the CoAE-MLSim approach is generated using Ansys Fluent (Fluent 2013), except for few cases where publicly available data sets are used. As stated earlier, the data requirements are minimal and each case requires about 100-1000 solutions, depending on the complexity of physics and di- mensionality of PDE conditions. The network architectures and training mechanics are general and described in appendix A.I
4.1 STEADY-STATE: INDUSTRIAL USE CASE OF ELECTRONIC-CIIP THEERAL COOLING
This experiment demonstrates the steady-state CoAE-MLSim approach. It consists of electronic chip package surrounded by air and subjected to heat sources with random spatial distributions due uncertainty in electrical heating. The temperature distribution the chip is governed by a natural convection process, which a balance between heating due to heat source and cooling because of flow. The heat sources are sampled from a Gaussian mixture model (up 25 Gaussians with random mean and variances) and represent high dimensional space (4096) with large variations, thereby making it incredibly hard to generalize across. The training data for this case corresponds only 300 solutions generated for random heat sources using Ansys Fluent.
Huat
Valosdy
gl (nerad)
Kutfies
cumar
Zurkibru
ocaut
Frand
-M0
605
410
000
HBC Am
Figure 7: Comparisons: Contour (left) & Line (right, red: CoAE-MLSim & blue: Fluent)
In Fig.
show the temperature and velocity magnitude contour and line plot comparisons between Ansys Fluent and CoAE-MLSim approach for an unseen heat source with very good agree- ment. Additional details provided in appendix section A.2.3
4.2 TRANSIENT: VORTEX DECAY
Ly eror vs time
Timestep 10
Timestep 30
Timestep 50 Truth CoAE-MLSim
Truth
Simm
Truth CoAE-MLSim
03
Eatrapabtten
33
Figure 8: Vortex decay over time with CoAF-MLSim approach
This experiment demonstrates the transient CoAE-MLSim approach. In this case, the transient dy- namics of flow modeled for different choices of initial vorticity. The training data (generated by Li al. (2020a)) used to train the CoAE-MLSim autoencoders comprises 500 initial conditions and first 25 timesteps. The testing is carried out 50 unseen initial conditions and extrapolated

---

Under review as a conference paper at ICLR 2022
up to 50 timesteps. It may be observed in Fig. g that the CoAE-MLSim predictions match well with the ground truth and the error accumulation is acceptable and significantly smaller than base- lines such UNet (Ronneberger et al.) 2013) and Fourier neural operator (FNO) (Li et al. (020a)), especially in the extrapolation region for an unseen test sample. This is attributed to the flux conser- vation autoencoder, which provides better control over solution trajectory in each time integration. Additional details are provided in appendix section A.2.5
4.3 DISCUSSION
Analysis for different subdomain sizes:
400
We test the performance of our approach for varying subdomain resolutions. The autoencoders in the CoAE-MLSim approach are trained using the data corresponding to the case 4.1 with subdomain resolutions of 83, 16 and 323. The trained models are used to solve for PDE solutions for 50 different unseen heat sources and their results, shown in the table below, are compared with Ansys Fluent on 3 metrics, Error in maximum temperature (hot spots chip), Loo error in temperature, Error in heat flux (temperature gradient) on the chip surface. Table [ shows results for 3 randomly chosen cases and Fig. plots the solve time averaged over all test cases.
Time
8'
16a
32
Subdomain size
Figure 9: Computation time for different sub-domain sizes
Case ID
Lool 83
T true
T pred
Err. Crr.(Tmon (Tmaz)
Avg.
Error
flux)
163
32
83
163
323
83
163
323
553
16.93
20.36
16.90
8.35
5.09
5.50
0.65
1.26
0.10
555
19.45
14.35
12.89
-4.32
-1.04
-1.99
0.64
0.59
0.09
574
20.1
17.80
10.20
12.20
8,90
5.00
1.97
1.86
0.13
Table 1: Analysis of different subdomain sizes
The accuracy very similar for different subdomain sizes, but the computational time is drastically different. Higher subdomain resolution corresponds to fewer subdomains in the entire domain and hence reduction in computational cost. The reduction in solve time is not linear because the latent vector compression is smaller for larger subdomains to represent a larger space of PDE solutions.
Stability:
(A)
A long standing challenge in the field of numerical simulation is to guar- antee the stability and convergence of non-linear PDE solvers. How- ever, we believe that the denoising capability of autoencoders (Vincent let al., 2010, Goodfellow et al.] 2016 Du et al., 2016 [engio et al.) 2013] Ranzato et al.] (207)) used in our iterative solution algorithm presents unique benefit, irrespective of the choice of initial conditions. In this work, we empirically demonstrate the stability of steady-state ap- proach for case 4.1 In scenario A, we randomly sample 25 initial so- lutions from uniform distribution and in scenario B we sample from 10 different distributions, such as Gumbel, Beta, Logistic etc. The mean convergence trajectory and the standard deviation bounds plotted in Fig. 10 show that the Ly norm of the convergence error falls is acceptable for all cases and demonstrates the stability of our approach.
0.995
(B)
Paause
Figure 10: Robustness of CoAF-MLSim approach
We observe that the CoAE-MLSim approach is about 40-50x faster in the steady-state cases and about 100x faster in transient cases as compared to commercial PDE solvers such as Ansys Fluent for the same mesh resolution in all the experiments presented in this work. Both CoAB- MLSim approach and Ansys Fluent are solved on a single Xeon CPU with single precision. We expect our approach to scale to multiple CPUs as traditional PDE solvers but single CPU compar- isons provided here for benchmarking. Moreover, code, whereas Ansys Fluent an optimized, C language pre-compiled code. We expect the C/C++ version of our algorithm to further provide independent speedups (not included in current estimates).
Computational speed:
algorithm is python language interpreted
8

---

Under review as a conference paper at ICLR 2022
Extent of generalization: Like other ML-based approaches, approach also operates within certain bounds of generalization with respect to the high dimensional space of PDE conditions. In many cases, the PDE conditions also have sparse representations, which makes generalization tougher. Here, we have demonstrated that our approach can generalize within the space of high- dimensional and sparse PDE conditions without compromising on computational speed and solution accuracy while using limited training data for autoencoders.
Coupling with commercial PDE solvers using solution initialization:
In experiment 4.1 we initialize solution variables in iterative al- gorithm with solutions computed by commercial PDE solvers demonstrate that our approach can be used as an accelerator to these solvers. In Fig. I we show that the coupling between our itera- tive algorithm and PDE solvers can result in significant convergence speedup for high fidelity solutions as compared zero initializa- tion. More details on this are in the appendix section A.2.3
Jan kond Soatee
Henoo Boy
2281220111 Sokien
10
20
30
40
ittentionn
Figure 11: Error history
Comparison with other ML baselines: Although our method is significantly different from traditional MI. methods, we have pro- vided comparisons of experiments from our paper and appendix with MI. baselines such as UNet (Ronneberger et al. 2015) and FNO (Li et al., 2020a), wherever available. The results are shown in the table 2 All the errors are computed with respect to results computed by a traditional PDE solver. The comparisons
carried over 50 unseen test cases. In the chip cooling experiment 4.1 the Log error in temperature is compared as it is the most relevant metric in this experiment. For the vortex decay case [a2e the mean absolute error is averaged over 50 timesteps. In both cases, the CoAE-MLSim performs better than the baselines and has small training parameter space. Since the CoAE-MLSim approach is unsupervised, local and ower-dimensionall it requires lesser amount of data. For consistency, we have used the same amount to train the ML baselines.
Table Comparison with baselines
Experiment
CoAE-
UNet
FNO
# params 400 K
ML.im
CoAE-MLSim
Chip cooling (Temperature)
20.36
117.07
X
UNet
7.418 M
Vortex decay
0.04
0.08
0.09
FNO
465 K
Laplace Eq. (in Sec. A.2.I
0.007
0.195
0.165
CONCLUSION
In this work, we introduced the COAE-MLSim approach, which is an unsupervised, low-dimensional and local machine learning approach for solving PDE and generalizing across a wide range of PDE conditions randomly sampled from high- dighndimensional distribution. Our approach is inspired from strategies employed in traditional PDE solvers and adopts an iterative inferencing strategy to solve PDE solutions. It consists of several autoeneoders that can be easily trained with very few training samples. The proposed approach is demonstrated to predict accurate solutions for a range of PDEs and generalize across sparse and high dimensional PDE conditions.
Broader impact and future work: In this work we aim to combine the ideas developed by tradi- tional PDE solvers with current advancements in machine learning and computational hardware and achieve faster simulations for industrial use cases that can range from weather prediction to drug discovery. Although the proposed ML-model can generalize across wide range of PDE condi- tions, but extrapolation to PDE conditions that are significantly different still remains challenge. However, this work takes big step towards laying down the framework on how truly generalizable ML-based solvers can be developed. In future, we would like to address these challenges of gener- alizability and scalability by training autoencoders on random, application agnostic PDE solutions and enforcing PDE-based constraints in the iterative inferencing procedure. In this work, we also demonstrate the potential of a hybrid solver by coupling with traditional PDE solvers. This will be investigated further along with extensions to inverse problems and scale invariance of PDE solu- tions. Finally, one of the limitations of this work is that it cannot handle unstructured meshes and this will be addressed in a follow-up paper.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 136-145. PMLR, 2017.
Anima Anandkumar, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differ- ential equations. In ICLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2020.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 19X 2019.
Deep equilibrium models.
arXiv preprint
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders, arXiv preprint 2rXiv:000..05991, 2020.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven dis- cretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116631):55344-5544 2019.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18, 2018.
Alex Beatson, Jordan Ash, Geoffrey Roeder, Tianju Xue, and Ryan Adams. Learning composable energy surrogates for pde order reduction. Advances in Neural Information Processing Systems, 33, 2020.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. arXiv preprint arXiv:1305.6665 2013.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduc- tion and neural networks for parametric pdes. arXiv preprint arXiv:2005.03/880 2020.
Mathis Bode, Michael Gauding, Zeyu Lian, Dominik Denker, Marco Davidovic, Konstantin Klein- heinz, Jenia Jitsev, and Heinz Pitsch. Using physics-informed enhanced super-resolution gener- ative adversarial networks for subfilter modeling in turbulent reactive flows. Proceedings of the Combustion Institute, 38(2):2617-2625,
7-2625,, 2021.
Shengze Cai, Zhicheng Wang, Frederik Fuest, Young Jin Jeon, Callum Gray, and George Em Kar- niadakis. Flow over an espresso cup: inferring 3-d velocity and pressure fields from tomographic background oriented schlieren via physics-infe informed neural networks. Journal of Fluid Mechan- ics, 915, 202
Shengze Cai, Zhicheng Wang, Sifan Wang. Paris Perdikaris, and George Karniadakis. Physics- informed neural networks (pinns) for heat transfer problems. Journal of Heal Transfer, 2021b.
James P Crutchfield and McNamara. Equations of motion from data series. Complex systems, 1(417-452): 1987.
Filipe de Avila Belbute-Peres. Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End- to-end differentiable physics for learning and control. Advances in neural information processing systems, 31:7178-7189, 2018.
Bo Du, Wei Xiong, Jia Wu, Lefei Zhang, Liangpei Zhang, and Dacheng Tao. Stacked convolutional denoising auto-encoders for feature representation. IEEE transactions on cybernetics, 47(4); 1017-1027, 2016.
10

---

Under review as a conference paper at ICLR 2022
Vikas Dwivedi, Nishant Parashar, and Balaji Srinivasan. Distributed learning machines for solving forward and inverse problems in partial differential equations. Neurocomputing, 120:299-316, 2021.
ANSYS Fluent. Ansys fluent. Academic Research. Release, 14, 2015.
Kai Fukami, Taichi Nakamura, and Koji Fukagata. Convolutional neural network based hierarchi- cal autoencoder for nonlinear mode decomposition of fluid field data. Physics of Fluids, 32(9): 0951 10, 2020.
Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: physics-informed geometry-adapiin convo- lutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428: 10079, 2021.
Frederic Gibou, Ronald Fedkiw, and Stanley Osher. A review of level-set methods and some recent applications. Journal of Computational Physics, 353:82-109, 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Ehsan Haghighat and Ruben Juanes. Sciann: A keras/tensorllww wrapper for scientific computa- tions and bhysies-informed deep learning using artificial neural networks. Computer Methods in Applied Mechanics and Engineering, 373:1 13552, 2021.
Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods Applied Mechanics and Engineering, 379:113741, 2021.
Haiyang He and Jay Pathak. unsupervised learning approach to solving heat equations on chip based auto encoder and image gradient. arXiv preprint 2xXv:20077009684, 2020.
Oliver Hennigh, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaustubh Tangsali, Max Rietmann, Jose del Aguila Ferrandis, Wonmin Byeon, Zhiwei Fang, and Sanjay Choudhry. Nvidia simnet" [TM]: an ai-accelerated multi multi-physies simulation framework. arXiv preprint arXiv:20/2.079::3 2020.
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning control pdes with differentiable physies. arXiv preprint arXiv:2001.025 100455, 2020.
Jonathan R Holland, James D Baeder, and Karthikeyan Duraisamy. Field inversion and machine learning with embedded neural networks: Physics-consistent neural network training. In AIAA Aviation 2019 Forum, pp. 3200, 2019.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Kariiadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.
Ioannis G Kevrekidis, C William Gear, James M Hyman, Panagiotis Kevrekidid, Olof Runborg, Constantinos Theodoropoulos, al. Equation-free, coarse graiieed multiscale computation: En- abling mocroscopic simulators to perform system-level analysis. Communications in Mathemati- cal Sciences, 144):715-762, 2003.
Ehsan Kharazmi, Zhongqiang Zhang, and George Karniadakis. hp-vpinns: Variational physics- informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering. 374:1 175:1, 2021.
Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In Computer Graphics Forum, volume 38, pp. 59-70. Wiley Online Library, 2019.
Dmitrii Kochkoy, A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. arXiv preprint arXiv:2102.01010,
2021.
11

---

Under review as a conference paper at ICLR 2022
Li Li, Stephan Hoyer, Ryan Pederson, Ruoxi Sun, Ekin D Cubuk, Patrick Riley, Kieron Burke, et al. Kohn-sham equations as regularizer: Building prior knowledge into maahhne--eaame physics. Physical review letters, 126(3):036401, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 2020a.
Zongyi Li, Nikola Kovachki. Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXiv:2006.0955 2020b.
Lu Lu, Haiyang He, Priya Kasimbeg, Rishikesh Ranade, and Jay Pathak. One-shot learning for solution operators of partial differential equations. arXiv preprint arXiv:2104.0551,, 2021a.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, ((3):218-22) 229, 2021b.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: deep learning library for solving differential equations. SIAM Review, 63(1):208-228 2021c.
Amir Maleki, Jan Heyse, Rishikesh Ranade, Haiyang He, Priya Kasimbeg, and Jay Pathak. Geom- etry encoding for numerical simulations. arXiv preprint arXiv:2104.07792, 2021.
Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 1 12789, 2020.
Romit Maulik, Bethany Lusch, and Prasanna Balaprakash. Reduced-order modeling of advection- dominated systems with recurrent neural networks and convolutional autoencoders. arXiv preprint arXiv:2002.004700 2020.
Takaaki Murata, Kai Fukami, and Koji Fukagata. Nonlinear mode decomposition with convolutional neural networks for fluid dynamics. Journal of Fluid Mechanics, 882, 2020.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidani. Efficient training of physics- informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 2021.
Ravi Patel, Nathaniel Trask, Mitchell Wood, and Eric C Cyr. physics-informed operator re- gression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 373: 13500, 2021.
Tobias Pfaff, Meire Fortunato, Alvaro based simulation with graph networks. arXiv preprint arXiv:2010.03409 2020.
and Peter W Battaglia. Learning mesh-
Gavin D Portwood, Peetak P Mitra, Mateus Dias Ribeiro, Tan Minh Nguyen, Balasubramanya Nadiga, Juan A Saenz, Michael Chertkov, Animesh Garg, Anima Anandkumar, Andreas Dengel, et al. Turbulence forecasting via neural ode. arXiv preprint arXiv:1911.05188. 2019.
Elizabeth Qian, Boris Kramer, Benjamin Peherstorfer, and Karen Willcox. Lift & learn: Physics- informed machine learning for large-scale nonlinear dynamical systems. Physica D: Nonlinear Phenomena, 406: 132401, 2020.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-141, 2018.
Maziar Raissi, Paris Perdikaris, and George E Kariiadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 378:686-707, 2019.
Rishikesh Ranade, Chris Hill, Haiyang He, Amir Maleki, and Jay Pathak. A latent space solver for pde generalization. arXiv preprint arXiv: 204::22004.0252.. 2021a.
12

---

Under review as a conference paper at ICLR 2022
Rishikesh Ranade, Chris Hill, and Jay Pathak. Discretizationnet: A machine-learnnng based solver for muvier-stoker equations using finite volume discretization. Computer Methods in Applied Mechanics and Engineering, 378: 13722, 2021b.
Mare Ranzato, Christopher Poulmey, Sumit Chopra, Yann LeCun, al. Efficient learning of sparse representations with an energy- mnrgy-bbseed model. Advances in neural information processing systems, 19:1137, 2007.
Chengping Rao, Hao Sun, and Yang Liu. Physics-informed deep learning for incompressible laminar flows. Theoretical and Applied Mechanics Letters, 10(3):07--22.. 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Alvaro Sanchez-Gonzales Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 8459-8468. PMLR, 2020.
Khemraj Shukla, Ameya D Jagtap, and George Em Karniadakis. Parallel physics-informed neural networks via domain decomposition. arXiv preprint arXiv:2104.10011 2021.
Anand Pratap Singh, Karthikeyan Duraisamy, and Ze Jía Zhang. Augmentation of turbulence models using field inversion and machine learning. In 55th AIAA Aerospace Sciences Meeting, pp. 0993, 2017.
Ehsan Taghizadch, Helen M Byrne, and Brian D Wood. Explicit physics-informon neural networks for non-linear upscaling closure: the case of transport in tissues. arXiv preprint arXiv:2104.01476, 2021.
Marc A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
Kiwon Um, Philipp Holl, Robert Brand, Nils Thuerey, et al. Solver-in-the-lopp Learning from differentiable physics to interact with iterative pde-solvers. arXiv preprint arXiv:2007.00016, 2020.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynam- ics from scratch towards fast, differentiable fluid models that generalize. arXiv preprint arXiv:2006.08762. 2020.
Hengjie Wang, Robert Planas, Aparna Chandramowlishwaran, and Ramin Bostanabad. Train once and use forever. Solving boundary value problems in unseen domains with pre-trained deep learn- ing models. arXiv preprint arXiv:2104.1053 2021.
Wujie Wang. Simon Axelrod, and Rafael Gómez-BBobarelll, Differentiable molecular simulations for control and learning. arXiv preprint arXiv:2003.00863; 2020.
Steffen Wiewel, Byungsoo Kim, Vinicius C Azevedo, Barbara Solenthaler, and Nils Thuerey. Latent space subdivision: stable and controllable time predictions for fluid flow. Computer Graphics Forum, volume 39, pp. 15-25. Wiley Online Library, 2020.
Jin-Long Wu, Heng Xiao, and Eric Paterson. Physics-informe machine learning approach for augmenting turbulence models: A comprehensive framework. Physical Review Fluids, 3(7); 074602, 2018
13

---

Under review as a conference paper at ICLR 2022
Tianju Xue, Alex Beatson, Sigrid Adriaenssens, and Ryan Adams. Amortized finite element analysis for fast pde-constrained optimization. In International Conference on Machine Learning. pp- 10638-10647. PMLR, 2020.
14

---

Under review as a conference paper at ICLR 2022
A APPENDIX
In the appendix, we provide additional details to support and validate the claims established in the main body of the paper. The appendix section is divided into 3 sections. In Section A.. provide details related to the network architectures, training mechanics and data generation for the autoencoders used in the CoAE-MLSim approach. In Section A.2 provide additional details and results for the experiments described in the main paper. Additionally, we have demonstrated the CoAB-MLSim different training the various autoencoders in the CoAE-MI Sim approach.
In Section A.3 we present details for reproducibility and for
A.1 NETWORK ARCHITECTURE, TRAINING MECHANICS AND DATA GENERATION
The training portion of the CoAE-MLSim approach proposed in this work corresponds training of several autoencoders to learn the representations of PDE solutions, conditions, such as geometry and PDE source terms as well as flux conservation and time integration. The boundary conditions are manually encoded using the strategy explained in Section A.1.3] We train all the autoencoders with the NVIDIA Tesla V- 100 GPU using TensorFlow. The autoencoder training one-time cost and is reasonably fast. In this section, describe details related to the network architectures for the different autoencoders, as well as training mechanics and data generation.
A.1.1 GEOMETRY AUTOENCODER
The geometry autoencoder learns compress signed distance field (SDF) representations of geom- etry. Mathematically, the signed distance at any point within the geometry is defined as the normal distance between that point and closest boundary of object. More specifically, for x R and object(s) Ω C R", the signed distance field Φ(x) is defined by:
(1, 02)
Ω
<(x)
d(x, 02)
where, ((x) is the signed distance field for x E R and objects Ω C R Gibou et al. (2018). Maleki al. (2021) the same representation of geometry successfully demonstrate the encoding of geometries. In this work, we use CNN-based geometry encoder to encode SDF representation
Encoded latont voctor
Convolustioall layer
Regular cornection
layer
Irpuit SDF
Tremspoved Convolutional leyer
Prodictdd SDF
Figure 12: Network architecture for geometry autoencoder Maleki et (021))
of geometries for relevant use cases. The architecture of this network shown in Figure The geometry autoencoder has CNN-based encoder-decoder structure. The encoder compresses the SDF representation to a latent vector, 7y and the decoder reconstructs the SDF representation. In the context of CoAE-MLSin approach, trained geometry encoder is used to represent SDFs local subdomains with latent vectors. Section A.2.4 we present results to demonstrate the generaliz- ability of the autoencoder encode and decode unseen geometries subdomains.
15

---

Under review as a conference paper at ICLR 2022
Training data and mechanics: The training data pertaining to the geometry autoencoder is structed from random geometries. Arbitrary geometries on entire computational domains are gener- ated and their SDFs are computed. The computational domain is divided into subdomains and the parts of geometry SDFs associated to every subdomains is used as part of the training samples. Since on subdomains, representations of complicated and arbitrary geometries can be learnt accurately. Moreover, we require only 200-400 geometries on entire computational domains train the autoencoder on subdomains. The autoencoder is trained until a Mean Squared Error (MSE) of 1e- 6 or Mean Absolute Error (MAE) of 1e-3 3 is achieved on validation set. The latent vector length is of the lowest possible size that can result in training errors below these spec- ified thresholds and can be determined through experimentation. Geometry encoders used in all experiments described in this work are trained as mentioned above.
the training is carried
A.1.2 PDE SOURCE TERM AUTOENCODER
The PDE source term autoencoder learns to compress the spatial distributions of source terms each subdomain the computational domain into a latent vector, 7).- The source term autoencoder uses the same architecture as in Figure 12. except that the inputs and outputs are the source term distributions.
Training data and mechanics: The training data for source terms is generated entire compu- tational domains by sampling from a Gaussian mixture model, where the number of Gaussian's ss mean and variance of Gaussian's arbitrary and span over orders of magnitude. The source terms are divided into subdomains, which are used as training samples for the autoencoder. In this case, generate about 200-400 such source term distributions and train until the MSE or MAE of the validation set drop below their respective thresholds, le and 1e-3. The latent vector length is chosen such that the training errors are below the specified thresholds. Source term encoders used in relevant experiments described in this work are trained as mentioned above.
A.1.3 REPRESENTATION OF BOUNDARY CONDITIONS
The boundary condition encoders can be learnt using the same autoencoders described Figure However, in this work design manual encoding strategy to establish the latent vectors for boundary conditions. This is because, the choice of boundary conditions numerical simulations considered in this paper is very limited. The boundary condition encoding strategy described in On each face of a subdomain, the boundary condition encoding, 1)b, is a vector of size 2.
12
Figure 13
Face of a subdomain
BC cncoding of cach face
{BC type, BC value}
Figure 13: Strategy for BC encoding
The first element of this vector indicates the type of boundary condition, followed by the boundary condition value. In this work, 3 types of boundary conditions considered, namely Dirichlet, Neumann and Open boundary condition and are indexed as 0, 1, respectively. Open boundary type refers faces that do not have any boundary condition imposed on them and suitable for faces of interior subdomains. For example, subdomain placed on the inlet boundary, such that the left boundary of the subdomain aligns with the inlet boundary of the entire computational domain, will have left face encoding of O, 0.5, where 0 refers to Dirichlet boundary and 0.5 is the BC value. As we move to applications in structural mechanics in future, new methods for encoding boundaries will be introduced.
A.1.4 PDE SOLUTION AUTOENCODER
Figure shows the network architecture used for encoding the solutions, 77, of all PDE solution vari- ables subdomains. It may be observed that the PDE solution autoencoders are also conditioned
16

---

Under review as a conference paper at ICLR 2022
on the geometry, source term and boundary latent vectors, that associated to the subdomains. Since, the PDE solutions are dependent and unique to PDE conditions, establishing this explicit dependency in the autoencoder improves robustness. We recommend the training of each solution variable using a different autoencoder to determine a latent vector which is independent the latent vector of other solution variables. This strategy of decoupling has shown increase the accuracy of the solution autoencoders. The specific parameters used in the network architecture can vary based on the size of each subdomain, complexity of physics and extent of compression achieved.
Encoded latent vector
Convolutional layer
Regular
Strided Comvolutional layer
Imput POE solutions
Transposed Convolutional layer
Predicted PUE solutions
Geometry vector
Boundary latent vector
Source term latent vector
Figure 14: PDE solution autoencoder
Training data and mechanics: The PDE solution variables very specific to the PDEs being solved and the engineering application being modeled. As a result, different PDE solution autoen- coder needs to be trained, when the application and the corresponding PDE is changed. Similar to the other autoencoders, the PDE solution autoencoder is trained on subdomains until an MSE of 1e-6 MAE of 1 is achieved on validation set. The compression ratio is selected such that the solution latent vector has the smallest possible size and yet satisfies the accuracy up to these tol- erances. Finally, the data for training these autoencoders is generated by running CFD simulations on entire computational domains for arbitrary PDE conditions. The generated solutions are divided into subdomains and used for training the PDE solution autoencoder. Since, the learning is local the number of solutions required to be generated are about 100-1000 for different PDE conditions. The solution autoencoders used in all the experiments have been trained with the strategy described above.
A.1.5 FLUX CONSERVATION AUTOENCODERS
The flux conservation autoencoder learns the local consistency conditions for group of neigh- borhood subdomains in the latent space. Each subdomain is characterized by PDE solutions and conditions and each of these affects the flux conservation autoencoder. As result, the inputs and outputs of this network are the latent vectors of solution (n)) geometry (7g)), source term (1%) and boundary (77)
group of neighborhood subdomains. All the solution variables of system of PDEs are stacked together with PDE condition latent vectors and the learnt using the autoencoder architecture shown in Figure 15 This autoencoder implicitly learns represent consistency condi- tions between neighboring subdomains. Since this autoencoder is only trained on locally consistent subdomains with continuous solutions across intersecting faces, tries to establish this consistency in neighborhood subdomain solutions for arbitrary inputs. Subdomains at the boundaries may have fewer neighbors and we propose 2 ways to handle this. Firstly, the information related to the missing neighbors can be substituted with vector of zeros. This would enable learning of all neighboring subdomain combinations with the same flux conservation network. Conversely, different flux con- servation networks be trained for subdomains with different number of neighbors. For example, a subdomain in the corner will have only neighbors, while an interior subdomain has neighbors. In this case, the interior and corner subdomains can be trained separately with different networks.
17

---

Under review as a conference paper at ICLR 2022
In experience, both approaches work equally well but have adopted the approach of zero padding in this work. The specific parameters used in the network architecture can vary based on the size of each subdomain, complexity of physics and extent of compression achieved.
Regular Connection
Dense Fully connected layer
17 on neighboring subdomains
1g neighboring subdemains
7/s on neighboring subdomains
Th on neighboring subdomains
Figure 15: Flux conservation autoencoder
Training data and mechanics: The training data generated for training the PDE solution autoen- coders is used to train these networks as well. 'The data is pre-processed such that groups of neigh- boring subdomains are collected together and the solutions and conditions associated with them are encoded using pre-trained autoencoders described in previous sections. This processed data is used for training the flux conservation autoencoder. This autoencoder is trained with an MSE loss and the training stops when the validation loss goes below 1 The flux conservation autoencoders used in all the experiments have been trained with the strategy described above.
A.1.6 TIME INTEGRATION NETWORKS
The network architecture and training data generation and mechanics are very similar to the flux conservation autoencoder. The only difference is in the network architecture, where the time in- tegration networks use latent vectors of PDE solutions at time t and conditions of neighborhood subdomains as the input but only predict the solution latent vectors of all solution variables on the center subdomain of the group of neighborhood subdomains at time t+1. Each PDE solution vari- able can be trained with a different time integration networks. The time integration networks used transient PDE related experiments have been trained with the strategy described above.
A.2 EXPERIMENTS ADDITIONAL RESULTS
In this section, we provide more results and details of the experiments discussed in the main paper. We have also demonstrated
A.2.1 STEADY STATE: LAPLACE EQUATIONS
The Laplace equation is defined as follows:
V = 0.0
(2)
subjected to Dirichlet boundary condition, Φ(2%) fo Neumann boundary condition,
fo.
18

---

Under review as a conference paper at ICLR 2022
Domain Boundary Conditions
FDM Solution
CoAE- MLSim
UNet-
FNO
FCNN Baseline
Based Baseline
6-15 6--1
p-1
1/40
May
Figure 16: Laplace equation contour results
The Laplace equation is second order, linear PDE that represents canonical problem for bench- marking linear solvers. It is solved on square computational domain with a resolution of 64x64 computational elements for random boundary conditions sampled from Dirichlet and Neumann. Here, we provide visual comparisons between the CoAB-MLSim approach and a second-order fi- nite difference method (FDM) approach in Fig. 16 The PDE solutions are locally scaled with the Dirichlet boundary condition. We compare the CoAE-ML with different ML approaches,, Unet Ronneberger et al. 2015), FNO (Li et al.) 2020a) and FCNN. The training data CoAE-MLSim corresponds to 300 solutions generated for arbitrary boundary conditions using second order finite difference method, whereas all the baselines are trained with 6400 solutions. The baseline models described below:
UNet: The input to this model is a two-channel grid size 64 x 64. The first channel captures the boundary condition encoding on 2D dimensional grid. There are boundary conditions chosen for this experiment, Dirichlet and Neumann. On a grid of zeros everywhere, boundaries are coded by replacing zeros with either ((Dirichlet) to the first channel, second channel is also a grid of zeros, and the edges' zeros are replaced with the magnitude of the boundary condition. The model is trained output the solution again on grid of 64 64. The UNet has 6 convolutional blocks, 2 at each nown-sampled size. The bottleneck size 8 The output of the bottleneck is again up-sampled in the usual fashion by concatenating the corresponding down-sampled output. The total number of learnable parameters in UNet baseline is equal 1.946 million.
2(Neumann) based on the boundary condition. Similar
FNO:
For the Fourier Neural Operator method as well, we have used the same input as in Unet. The FNO model is same as the original implementation Li et al. (2020a). The FNO model has 1.188 million parameters.
In this model, we consider the boundary condition encoding as the input as opposed to representation of the boundary condition as grid. This network consists of fully connected network and a convolutional neural network. The boundary condition encoding is first transformed into vector of size 1024. This vector is then reshaped into 32 32 grid. The grid is then passed to convolutional network and the solution is then transformed to 64 64 grid. The model has 14.8 million learnable parameters.
FCNN:
may be observed from Fig. 16 that the CoAE-MLSim approach outperforms the baseline ML models. The mean absolute errors for the CoAF-ML approach over 50 unseen testing samples
19

---

Under review as a conference paper at ICLR 2022
is 7 On the other hand, Unet, FNO and FCN have mean absolute errors of 1.65c 1.9e-1 and 3e respectively. All errors are with respect to the second order FDM solution. It may be observed that the FCNN performs better than both UNet and FNO and this points to an important aspect about representation of PDE conditions and its impact on accuracy. The representation of boundary conditions 64x64 grid is very sparse and high-dimensional, making it very challenging for the networks to learn. On the other hand, the FCNN uses low-dimensional, dense encoding as an input and hence able to learn more effectively. Nonetheless, the CoAB-MLSim approach provides the best performance.
For the CoAE-MLSim approach. the computational domain is divided into 64 subdomains such that each subdomain has a resolution of 8x8 elements. The 64 dimensional PDE solution is represented by latent vector of size using a CNN-based autoencoder. The total number of parameters in the solution and flux autoencoders are 130, 000. The boundary conditions on each subdomain are encoded using the representation provided in Section A.1.3] Since, this is a steady state problem, the CoAE-MLSim iterative solution algorithm is initialized with a solution field equal to zero in all test cases.
A.2.2 STEADY STATE: DARCY EQUATIONS
The Darcy equation is defined as follows:
VV.(aV(())= =
(3)
is subjected to different diffusivity conditions, a(T) f(Ø) in a 2-D bounded computational domain between (0, 1). The problem setup as well the data to train the autoencoders is taken from Li et al. (2020a). We use about 400 samples for training the flux conservation autoencoders, and 50 randomly picked samples for testing from the remaining data. The computational domain in this problem is 2D and has resolution of 241x241. The domain is divided into 241 subdomains of resolution 21x21 each. The solution on each subdomain is encoded into latent vector of size 47 and the diffusivity is encoded into size 21. Since, this is steady state problem, the CoAE-MLSim iterative solution algorithm is initialized with a solution field equal to zero iterative algorithm convergence tolerance is stringent and set to L2 le-7.
test cases. The
The main purpose of including this numerical experiment in the paper to present an ablation study that evaluates the effect of autoencoder bottleneck layer size on the accuracy and computational speed of the solution algorithm used in the CoAE-MLSim approach. As our approach is based on autoencoders have rough idea that there must exist optimum latent vector size which would provide the highest accuracy and a reasonable computational cost during inference time. However, in this experiment we validate this hypothesis. Here, present results from our experiments to analyze the effect of ffux-conservation autoencoder bottleneck layer size on the generalizability. accuracy, and performance of our approach. We choose the flux-conservation autoencoder for this analysis because it is the primary workhorse of our approach and is evaluated repeatedly during the iterative inferencing. We perform experiments with bottleneck layer sizes, 16, 32, 64, 128, 256 for an input size of 340.
The results from these experiments are presented the table below:
Bottleneck Size (Compres- sion ratio)
Testing error
Convergence iterations
Solve time
8 (42.5)
0.0241
15
0.23
16 (21.25)
0.0107
37
0.57
32 (10.62)
0.0067
45
0.706
64 (5.32)
0,0081
55
0.85
128 (2.65)
0.133
140
2.18
256 (1.32)
0.198
168
2.73
Table 3: Ablation study for different layer sizes
20

---

Under review as a conference paper at ICLR 2022
The testing error formulation is shown in Eq. 4
L2 prrop Yerue)
Lz((Yruc))
may be observed from Table 13 that as the compression ratio the flux conservation autoencoder decreases, begins overfit and the testing error well the number of convergence iterations and computational time significantly increase. On the other hand, if the compression ratio is too large the testing bottleneck size compression ratio where the best testing error is obtained and the computational time is not too large.
increases. For all autoencoders used in this work, there exists optimum
A.2.3 STEADY STATE: ELECTRONIC CHIP THERMAL PROBLEM
This is industrial use case where the domain consists of chip, which is sandwiched in between an insulated mold. The chip-mold assembly is held by a PCB and the entire geometry is placed inside fluid domain. The geometry and case setup of the electronic chip cooling observed in Fig. 17 The chip is subjected to electric heating and the uncertainty in this process results in random spatial distribution of heat sources the on the surface of the chip. Fig. shows an example of the various distributions of heat sources that the chip might be subjected to due to electrical uncertainty.
be
Fluid
Chip package
Support
Zoomed in chip
Chip
Mold
Substrate
PCB
Figure 17: Electronic chip cooling geometry
Figure 18: Different power map distributions
The physics in this problem is natural convection cooling where the power source is responsible for generating heat on the chip, resulting an increase chip temperature. The rising temperatures get diffused in to the fluid domain and increase the temperature of air. The air temperature induces velocity which in turn tries to cool the chip. At equilibrium, there a balance between the chip temperature and velocity generated and both of these quantities reach steady state. The objective of this problem is to solve for this steady state condition for an arbitrary power source sampled from Gaussian mixture model distribution, which extremely high dimensional with up 25
21

---

Under review as a conference paper at ICLR 2022
Gaussian's. 's, each with different mean and variance, on 4096 dimensional domain. The governing PDEs that represent this problem are shown in Eqs. 5
Continuity Equation:
vv=
1 42v -
Momentum Equation:
(v. + Vp
Re
(5)
Heat Equation in Solid:
V. (aVT) - =
Energy Equation in Fluid:
(v. - (QVT) =
where, V Uy; Mys Uy the velocity field 2, y, 2, p is pressure, is temperature, Re and are flow and thermal properties, P the heat source term, gT is the buoyancy term. is the spatially varying power source applied on the chip center. The main challenges are capturing the two-way coupling of velocity and temperature and generalizing over arbitrary spatial distribution of power.
The coupled PDEs with solution variables, v Ur, Myy Uz, P. T are solved on fluid and solid domain with loose coupling the boundaries. The fluid domain is discretized with 128 elements in the domain and the solid domain (chip) is modeled as 2-D domain with 64 elements as it is very thin in the third spatial dimension.
The data to train the autoencoders in the CoAE-ML approach is generated using Ansys Fluent and corresponds 300 PDE solutions. The computational domain divided into 512 subdomains, each with 16 computational elements. The solution autoencoders for the 5 solution variables are trained independently on to establish lower dimensional latent vectors with size on the subdomain level. The geometry and boundary conditions do not vary and hence autoencoder is not trained for them in this experiment. The source term autoencoder is trained using randomly generated power source fields. Figure 19 shows a few results of the reconstruction capability of the autoencoders. The source term latent vector has a size 39.
Ground Truth
Prediction
Ground Truth
Prediction
20
1s
465
20
Figure 19: Power source autoencoder for CoAE-MLSSm approach
In the main body the paper. Section 4.3 as well as in this section we compare our approach with other baseline ML models such as UNet (Ronneberger al. 2015) for this experiment and here briefly explain the network architecture used.
UNet:
The architecture of the UNet (Ronneberger al. 2015) is follows: Since the original
dimension of the power map is 64?, to construct a 3D chip power map, zero padding is applied top and bottom of the 2D power map, the input dimension of the Unet is 643. Each module
22

---

Under review as a conference paper at ICLR 2022
of the contracting path consists of 2 convolution layers, max pooling layer and dropout layer. The number channels changes from 1 to 48 in the contracting path and the module is repeated times. The module in the expansive path consists of 1 Upsampling layer, 1 concatenation layer and 2 convolution layers. Similar to that of the contracting path, the module is repeated times and the last layer is convolution layer with 1 filter. The activation function used is 'Relu', the convolution kernel size is 3 and the pooling window size is 2.
Additional results:
Here, we present comparisons between CoAE-ML approach and Ansys Fluent for 2 additional cases. We compare velocity magnitude and temperature on plane contorus well line plots. Also, we compare additional parameters obtained from the simulation such as temperature on PCB- Fluid interface, pressure the domain, total energy transfer between chip-fluid interface. It may be observed that the results of our approach agree well with commercial PDE solver and this continues to work for other choices of power sources from the Gaussian mixture model distribution. Since, this is steady state problem, the CoAE-MLSim iterative solution algorithm is initialized with a solution field equal to zero for solution variables all test cases.
Heat
Velecity on YZ. plane
Temperature on YZ plane (rcomed)
Temperature chip surface
Anes Fluent
23641
044w42
03
136441
020w00
52ee2
CoAE-MLSHH MSE-7
MSL-de
Figure 20: Contour plot comparisons of CoAE-MI Sim and Ansys Fluent for test case
X-axis through chip center
Y-axis through chip center
Z-axis through chip
center
600
E00
Temgersbure
520
500
500
400
600
0.00
300
-0.02
0.00
0.02
300
-0.30
0.35
0.15
Magnittte
0.00
0.30
0.0
-0.1
0.0
011
Figure 21: Line plot comparisons of CoAE-MLSim and Ansys Fluent for test 1 (red: CoAE- MI.im blue: Fluent)
Finally, this experiment provide further analysis of the CoAE-MLSim approach. The different sub-experiments are listed below:
Comparison against UNet for various test cases.
2. Analysis solution convergence during different iterations of the SoAE-MLSim approach
3. Analysis of coupling with commercial PDE solvers using solution initialization
23

---

Under review as a conference paper at ICLR 2022
Temperature PCB-Fluid interface
Pressure on XY plane
Total Energy on Chip-Fluid interface
Fiuent
Ansys
Ansys
Fluent
Fluent
E3twdi
CoAE-ML Sim
CoAF-MLSSm
CoAE-MLSin
Figure 22: Other parameter contour plot comparisons of CoAE-MLSim and Ansys Fluent for test case
Temperature YZ. (monmed)
Velocity YZ plane
chip
Angs Floret
4.0v42
200e01
13weh
(bod)
24
sowcs
32
COAF-MILS Sin
Scr
MSF
Figure 23: Contour plot comparisons of CoAE-MLSim and Ansys Fluent for test case 2
X-axis through chip center
Y-axis through chip center
Z-axis through chip center
600
600
500
500
400
300
-0.02
0.00
0.00
300
-0.02
0.02
0.35
0..5
0.30
0.30
0.05
0.05
0.2
30
-0.00
0.00
0.10
0.30
-0.0
0.20
Figure 24: Line plot comparisons of CoAE-MLSim and Ansys Fluent for test case 2 (red: CoAE- MLSim blue; Fluent)
Comparison of CoAE-MLSSm and Unet:
Since very few training samples train the CoAE-MLSim approach, it is important to demonstrate that our approach is not memorizing and that the physics represented by the experiments non-trivial. Hence, investigate this by comparing our approach with Unet (Ronneberger al., 2015). We use both methods to solve for 5 unseen power sources and the results obtained are compared with Ansys Fluent with respect to the metrics discussed below.
1. Error in maximum temperature in computational domain (hot spots chip).
24

---

Under review as a conference paper at ICLR 2022
Temperature PCB-Fluid interface
Pressure on XY plane
Total Emergy on Chip-Fluid Interface
Tee Froeer
Ansys
Fluent
Amsy
Anays
20twe
Fluent
Hdwll
2136/0
CoAE-MLSim
196-0
137w05
CoAE-MI.Shn
[Pe]
[N]
CoAE-MLSim
Figure 25:
Other parameter contour plot comparisons of CoAE-MLSim and Ansys Fluent for test
case
Case ID
Lo [ true
pred
Error (I (maas)
Aug. Siror[heat flux)
CoAE-MLSim
Unet
CoAE-MLSim
Unet
CoAE-MLSim
Unet
553
20.36
117.07
5.09
-117.06
1.26
7.41
554
10.50
27.24
3.40
27.25
1.45
0.93
555
14.35
62.08
-1.04
-62.08
0.59
2.76
575
15.60
195.52
-0.75
-195.51
0.80
15.78
574
17.80
74.72
8.90
-74.72
1.86
3.41
Table 4: Comparison of UNet and CoAE-MLSim
2. Too error in temperature in the computational domain,
3. Error in heat flux (temperature gradient) on the chip surface
These metrics are more suited for engineering simulations and provide a much better measure for evaluating accuracy and generalization than average based measures. The state-of-the-arr Unet (Ronneberger et al.) (2015) is trained on the same number of training samples as used by the CoAE- MLSim approach using the architecture described above. The results compared in the table below.
It may be observed that the approach outperforms the UNet for all unseen power sources.
Evolution of PDE solution during iterative inferencing:
Figure 26 shows the evolution of the CoAE-MLSim solution plane cut through the center of the chip and normal to the z direction, at different iterations until convergence. The results are shown for the y component of velocity for test case presented in Section 4.1 and compared with a converged Ansys Fluent solution for the same case. The initial solution provided to the solver sampled from uniform random distribution. At iteration conservation autoencoder denoises the random signals. In the following iterations, the solution begins to develop based on the source term encoding constraint and finally converges iteration 35. The stable progression of the solution points to the robustness and convergence of the CoAE- MLSim.
may be observed that the flux
Convergence speedup after coupling with traditional PDE solvers:
As mentioned earlier, in all the steady-state experiments carried out in this work, all the solution variables are initialized to zero in the CoAE-MLSim solution algorithm. In this experiment, we argue that better initialization of the solution algorithm can result in faster convergence and the better initialization can be obtained by coupling with commercial PDE solvers. To test this hypothesis, we generate 5 different initial conditions listed below and compare their convergence trajectories with the original case of zero initialization. In each case the solution corresponding to the specified resolution is computed using Ansys Fluent and interpolated on to the 128x 128x used by CoAE- MI. Sim approach.
25

---

Under review as a conference paper at ICLR 2022
Amay
Error
Anyy
CaAEMI-SSn
0.25
0.25
Iteration
Itecatinn 20
0.00
+0.00
0.25
0.25
Deration
Iecation 25
0.00
00.00
0.25
0.25
lirration
0.00
0.00
0.25
0.25
Iteration 15
0.00
0.00
Y
Figure 26: Convergence of CoAB-MLSim solution through the iterative procedure
(a) Initialization with solution
(b) Initialization with coarse resolution Ansys Fluent solution generated 32x32x32 mesh
(c) Initialization with medium resolution Ansys Fluent solution generated on 64x64x64 mesh
(d) Initialization with fine resolution Ansys Fluent solution generated on 128x128x 128 mesh
(e) Initialization with fine resolution Ansys Fluent solution generated on 128x 28x128 mesh with added random Gaussian noise
The convergence history comparison observed that with zero solution initialization, the CoAF-ML iterative solution algorithm takes about 41 iterations to converge. On the other hand, when initialized with the best possible initial guess, which is the ground truth solution on fine grid resolution, it converges add Gaussian random noise of 25 % maximum relative to the solution each computational element. With the added noise, the solution converges in about 8 iterations. Finally, when initialized with coarse grid solution generated Ansys Fluent on 323 and 64 resolutions, CoAE-MLSim take 27 and 11 iterations respectively. The convergence is still faster than zero solution initialization by 1.5x and 4x, respectively. Nonetheless, this experiment demonstrates that the convergence of the CoAE-MLSing approach can be improved with better initial guess and on the flip side our approach can be used an accelerator to commercial PDE solvers.
unseen test geometry is shown in Fig. 27 It may be
iterations. Next,
Analysis of train and test errors:
In this section, have carried on both training and testing PDE conditions. In this problem, we use 300 solutions for training the autoencoders and another 300 for testing. The PDE conditions with respect to both the training and testing set are used in the iterative solution algorithm generate solution. In table 5 we compare average results over the training set and the testing set for the infinity norm and error in maximum value of temperature in the computational domain. We also provide the best enror that we obtain from all the training samples.
experiment to compare the accuracy of the inference algorithm
may be observed that, the CoAE-MLSim approach performs same on the training set as would perform the testing set. The training portion of our model involves training several autoencoders. At inference time, the pretrained encoders cannot directly provide the solution directly for partic- ular setting of PDE conditions. We have the iterative algorithm to compute equilibrium
26

---

Under review as a conference paper at ICLR 2022
Zero Solution
Convergence error
32x32x32 Solution
64x64x64 Solution
-2
128x128x128 Solution
128x128x128 Solution with noise
-4
-6
10
20
30
40
Number of iterations
Figure 27: Convergence of CoAB-MLSim solution after coupling with commercial PDE solvers.
Temperature
Error (Tmaxx)
L
Best training error Average training error Average testing error
3.45
-0.3
13.6
2.78
15.1
3.93
Table 5: Comparison of training testing errors
solution for a specified set of PDE conditions. The iterative algorithm must figure out the solution and is never taught anything about the trajectory get there. As result, the algorithm can perform the same on the training PDE conditions as it would on the testing conditions. This is different from traditional ML methods, which work better in training than testing.
A.2.4 STEADY STATE: FLOW OVER ARBITRARY OBJECTS
In this use case, the CoAE-MLSime demonstrated for generalizing across a wide range of geometry conditions. The geometry of objects are represented with signed distance field representation and is extremely high- digh-dimensional (512 (512-4096). The use case consists of 3-D channel flow over arbitrarily shaped objects as shown Fig. 28 The domain has velocity inlet specified 1m/s and zero pressure outlet boundary conditions on 2 surfaces, while the rest of the surfaces are walls with no-slip conditions. The shape of the object immersed in flow is arbitrary and the objective is to demonstrate the generalization of the CoAE-MLSSmm for such geometric variations.
Velocity
3-D Computational Domain on cut plane
inlet
Pressure outlet
Object
Figure 28: Computational domain
27

---

Under review as a conference paper at ICLR 2022
The computational domain, in this case, is elongated in the direction, such that it consists of 320 computational elements in that direction, while the y and & directions have 160 and 80 computational elements each. The domain is decomposed into subdomains with 16 elements in each direction. Boundary conditions, geometry and PDE solutions are encoded on each subdomain using pre-trained encoders. The PDES solved in this problem consist of 4 solution variables, include 3 components of velocity and pressure. The governing equations corresponding to these variables are shown in Eq. The subdomain solution of each solution variable is compressed a latent vector size 31. The geometry is represented using signed distance fields, which are compressed to a latent vector of size 29. Boundary conditions are encoded using the boundary encoder described in Section A.1.3 There is no source term in this case and hence, it is excluded from the flux conservation autoencoder. The training data this case corresponds to 300-400 solutions generated for arbitrary geometries using Ansys Fluent. Since, this is steady state problem, the CoAF-MLSSm iterative solution algorithm is initialized with a solution field equal to zero for all solution variables in all test cases.
Continuity Equation:
vv=0 (v.V)x
Momentum Equation:
1
vv==
(6)
Re
where, v == uuy My. Ug is the velocity field in æ. y, 2, p pressure, is temperature, Re Reynolds number representing flow properties.
Figure 29 shows the velocity, pressure and wall flux contour plots comparisons between CoAE- ML approach and Ansys Fluent for flow around an unseen arbitrary object. The results match to an acceptable accuracy. In fact, the mean absolute error for pressure and velocity magnitude over 50 unseen test samples is 2.3e -2 and 9.4e
3-0 abject
X-Veletry
Pressure
Wall Flaxes
Anny
Flaeet
Flacot
CoAE-MESiO
MAE:
Figure 29: Flow over external object contour plot comparisons
Additional examples Figure 30 shows comparisons of CoAF-MI.Si with Ansys Fluent for un- seen objects in addition to the example shown in Figure 29 The solver generalizes well and the errors fall within an acceptable range.
Geometry encoder performance
Figure 31 shows the comparison between the true signed distance field and the reconstructed field from the geometry autoencoder. In this example, the geometry autoencoder is evaluated on each subdomain, but all subdomains are reassembled using their connectivity information to obtain the SDF on the entire computational domain. The contour plots in Figure 31 on planes cut through the center of the geometry along z direction. It may be observed that the autoencoder reconstructs agree well with the ground truth for unseen cases. The mean absolute error in all the cases presented here is less than 1
A.2.5 TRANSIENT: VORTEX DECAY FLOW
This is a similar use case from Li et al. (020a) where a 2-d Navier-Stokes equation for viscous, incompressible fluid in vorticity form on the unit torus. The PDE and case details corresponding to this may be found in [Ii et al. (2020a). In the main paper, we provided a single result for an unseen initial condition when integrated in time. Here, we will provide visual comparisons of additional results and additional details about the CoAE-MLSim approach used for this problem.
As described earlier, the mesh resolution considered in this experiment consists of 64x64 computa- tional elements. The dataset considered this experiment corresponds to a viscosity of 1e and contains transient evolution of 5000 initial solution fields over 50 timsesteps. The autoencoders of the CoAE-MLSim approach are trained with data corresponding to only 500 initial conditions and first 25 timesteps. The computational domain is divided into 64 subdomains of 82 resolution. The
28

---

Under review as a conference paper at ICLR 2022
3-D objects
X-Velecity
Pressure
Ansys
Fluent
Fluent
CoAF-MLSSm
MAE: 3e
CoAE-MILSim
Vile
Ansys Fluent
Ansys
Fluent
CoAR-ML
MAE:: 2e
CoAE-ML-Sim
Ansys Fluent
Ansys Fluent
MAE:
CoAE-MLSim
MAE:: Je
CoAE-ML
Amsys
Fluent
Fluent
Adedt
MAE: 9e
CoAE-MLSim
CoAE-MLSim
Figure 30: CoAE-MLSim vs Ansys Fluent comparisons for different objects
Truth
Prediction
Error
Truth
Prediction
Error
Figure 31: Geometry encoder generalization
solution autoencoder encodes vorticity on each subdomain into a latent vector of size 11. The flux conservation and time integration network are trained using the same data. The autoencoder net- works combine to have a total of 400, 000 parameters. The time integration network is designed such that the input contains solution latent vectors on neighboring subdomains of previous time steps to predict the latent vectors of the next time step. As result, all the test runs start from the 10th timestep. During each timestep integration during the solution inference, the flux conservation autoencoder is evaluated until iterative convergence is achieved, similar to the steady state version of the CoAE-MLSin approach and this convergence is achieved to specified tolerance of le in about 2-4 iterations. As a result, the total evaluation time might be slower than other ML-approaches but is still faster than commercial PDE solvers by around 100x and moreover, provides for a more stable and accurate transient dynamics.
29

---

Under review as a conference paper at ICLR 2022
well in this section compare approach with other baseline ML models such as UNet (Ronneberger et al. 2013) and Fourier Opertor Net (FNO) for this experiment and here briefly explain the network architectures used.
In the main body of the paper, Section 4.3
UNet: Our UNet architecture includes 18 convolutional layers with channel sizes of 10 (input), 64, 64, 128, 128, 128, 128, 256, 256, 256, 512, 384, 384, 256, 256, 192, 192, 128, 64, (output) with 9 before the bottleneck (encoder) and after the bottleneck (decoder). Skip connections were used connect layers before and after the bottleneck. In the encoder, the spatial resolution was reduced using four max-pooling layers with kernel size 2. In the decoder the original spatial resolution was recovered using four bilinear upsampling layers with kernel size The total number of learnable parameters is 7.418M.
FNO: Our FNO architecture was that of the original implementation (Li et al. 2020a). The total number of learnable parameters is 465k.
Next, we compare results for 3 unseen initial conditions. It may be observed that the CoAE-MLSim predictions match well with the ground truth data and the error accumulation is acceptable, espe- cially the extrapolation range. It may observed that our method outperforms FNO and UNet in terms of error accumulation.
Timestep
Timestop 30
Timestep
Truth
CoAE-MLSim
Tnuth
CuAE-MLSim
Truth CoAE-MLSim
Case 1
-04
Case 2
Case
Figure 32: Contour comparisons of vortex decay different time steps
Case
Case 2
Case
0.7
Coas ML5
0.5
Codz MLSin
Jite
Code MLSim
Uhat
0.6
L
0.4
Extrupolation
crror
Interpolation
0.2
0.2
0.10
0.1
0.1
0.05
0.0
0.0
0.00
10
15
20
25
30
35
40
45
10
15
20
25
30
35
40
45
10
15
20
25
30
35
40
Timesteps
Figure 33: Error accumulation at different time steps
Finally, we perform an additional study where average the predictions over 50 unseen testing samples from different initial conditions for each of the 50 timesteps. The results from this exper- iment are shown in Figure 34 It may be observed from the figure that our approach performs as good as FNO and UNet in the interpolation regime but outperforms both methods the extrapola- tion regime large margin.
Additionally, the table we provide the average errors for the approaches in the interpolation and extrapolation regions separately. For the interpolation regime the averages are computed 50 testing samples and until 25 timesteps, while for extrapolation regime they are computed over 50 testing samples and for timesteps between 25 and 50.
30

---

Under review as a conference paper at ICLR 2022
CoAE-MLSim
0.5
UNet
Extrapolation
L2
0.3
error
0.2
Interpolation
0.1
0.0
10
15
20
25
30
35
40
45
Timesteps
Figure 34: Error accumulation different time steps averaged over 50 unseen test samples
Bottleneck Size (Compres- sion ratio)
Testing error
Solve time iterations
Convergence
8 (42.5)
0.0241
15
0.23
16 (21.25)
0.0107
37
0.57
32 (10.62)
0.0067
45
0.706
64 (5.32)
0,0081
55
0.85
128 (2.65)
0.133
140
2.18
256 (1.32)
0.198
168
2.73
Table 6: Ablation study for different layer sizes
It may be observed that the CoAE-MLSim performs as well as FNO and UNet in the interpolation region but outperforms both of them for long time flow dynamics.
A.2.6 TRANSIENT: FLOW OVER A CYLINDER
Finally, we present demonstration of the CoAE-MLSim approach in solving the flow around cylinder problem in transient setting. The setup and geometry is similar to the presented in Section [.2.44 except that the flow Reynolds number much higher, equal to 200, in order to induce unsteady phenomenon in the flow, commonly known as vortex street. In this case, the CoAE- MLSim approach is trained on Reynolds number of 50 and 1000 and tested on 200. The timestep used for training is 20x larger than the one used by Ansys Fluent to generated the training and testing data. The complexity of the problem is increased by adding a constant heat flux to the cylinder, resulting in dissipation of temperature with the flow. The governing equations corresponding this case are presented in
Continuity Equation:
=vv= = 0 (v.V)v
8v
Momentum Equation:
Vp
v2v= Re
0t
0T
Energy Equation in Fluid:
(v. V. (QVT)) =0
at
where, v = Ugy uy Uy is the velocity field in x, y, z, t, p is pressure, temperature, Re and a are flow and thermal properties.
Figs. 35 and compare the vector plots on cut center planes for velocity magnitude and temperature with Ansys Fluent at different time steps. The CoAE-MLSim approach captures the transient flow dynamics with an acceptable enror in comparison to the Ansys Fluent. Furthermore, in Figure]37] we plot the total surface drag force of the cylinder function of time evaluate the vortex shedding
31

---

Under review as a conference paper at ICLR 2022
Ansys Fluent
0:20e
8.10e::
7.03ea
5,80e0
4,77e0
1
2
4 4.00-
2.00e
CoAE-MLSim
MSE=1e
MSE=2e
MSE=7e-6
9.8te00
8.17ed
474e0
3.6000
2,40+0
1330+
1
2:10a
Figure 35: Transient flow comparisons of velocity magnitude
frequency of the flow. The CoAB-ML MSm predictions satisfactory and continues to predict well as the solution progresses ahead in time.
A.2.7 TRANSIENT: BURGER'S EQUATIONS
The 1-d Burger's equation is shown in Eq
du
+ =
=
(8)
Ot
It is subjected different initial conditions, u()) (.) in 1-D bounded computational domain between (0, 1). The problem setup as well as the data to train the autoencoders is taken from [i et al. (2020a). The motivation of this experiment is to show the difference in performance between the two transient approaches mentioned in the paper: fully coupled and decoupled and validate the claim that the decoupled approach is better than the fully coupled approach in predicting long range time dynamics. The fully coupled transient approach is the one where space and time are coupled and in this example solved as a 2-D domain, where as the in the decoupled approach the time integration is handled separately from flux conservation as explained in the paper.
The 1-D computational domain has resolution of 512 computational elements. The computational case in the case of fully coupled case is a 2d domain with 512 elements x 50 timesteps. We use 500 solutions and 50 timesteps per solution to train the autoencoders both approaches. The I-d domain in the decoupled case is divided into 16 subdomains with a resolution of 32 computational elements each. On the other hand, the fully coupled case is divided into 48 subdomains with a resolution of 32 elements 32 timesteps. In both cases, the testing is carried out for 30 unseen samples integrated all the way up to 200 timesteps, which is more than twice outside the training regime. The relative errors, from Eq. g averaged over the testing samples for each timestep in
32

---

Under review as a conference paper at ICLR 2022
Ansys Fluent
48exd2
3 40u402
3 3.3e-0z
3.00w022
000+022
3 1000002
CoAE-MLSim
MSE=2e²
MSE=4e2
MSE=5e2
100e+022
3 544402
3 48cx02
3 3.40xx02
3.00xx0
3.18ee22
3 1eeee
3:00e402
3 000
[8]
Figure 36: Transient flow comparisons of temperature
90
Drag force
85
80
75
ML-Solver
Fluent
70
10
15
20
25
30
time (s)
Figure 37: Total drag force comparison
the extrapolation regime (after 50 timesteps) shown in Figure may be observed that the decoupled transient approach has a much smaller error accumulation than the fully coupled transient approach, therby validating our claim made in the paper.
A.3 DETAILS FOR REPRODUCIBILITY
In this section, provide the necessary details for training the CoAE-MLSim. As emphasized pre- viously, the CoAE-MLSim training corresponds to training several autoencoders for PDE solutions, conditions, such geometry, boundary conditions and source terms and for flux conservation. In
33

---

Under review as a conference paper at ICLR 2022
0.04
reloive etter
0.03
space decoupled tima-spaze coupled
0.02
0.01
0.00
time
Figure 38: Comparison between fully coupled and decoupled transient approaches
Figure we present flow chart of the steps that can be followed train each of these autoen- coders. The specific training details and network architectures may be found in Section A.1
For given set of coupled PDEs, start with 100-1000 sample solutions on a computational do- main with n, m, computational elements in spatial directions V. respectively. The solutions are divided into smaller subdomains of resolution, n/16, m/ m/16,1 p/16. Fach subdomain can has PDEs solutions and PDE conditions associated with it. The PDE solutions are used as training sam- the variables on the
ples to the PDE solution autoencoder to learn a compressed encoding subdomain. On the other hand, the autocncoders for PDE conditions can be trained with completely random samples, which may or may not be related to sample solutions. Once the PDE solution and condition autoencoders lution and PDE condition latent vectors on groups of neighboring subdomains are stacked together. These groups of stacked latent vectors used for training the flux conservation autoencoder. The trained PDE solution, condition and flux conservation autoencoders combine to form the primary components of the CoAE-MLSim.
trained, neighboring subdomains are grouped together and the
The solution algorithm of the CoAE-MLSim has been described in detail in the main body of the paper and may be used for solving for unseen PDE conditions.
Decompese "m"')) computational domains into (n/16"m/ p''6) subdensins
Smaller (6/16" p' subdomain solutions and their corrsponding goometry, source semm and houndary conditions
Learn PDE solutions on (u/16 (016"m/166pp166) subdommins using the solution Autocncoder.
Identify neighboring subdomains for all PDE
solutions condicion:
PDE condition encoder oetworks leantt on (n/16"m/166p/1) subdomains
Neighbbrr connectivity information is deteriined and stored
Desader PDE variables (latens space solution)
Encoder networks PDE variables (salution latent space)
Encude all neighboring sublomains for all PDE solutioes and conditions
Couple PDE solution condition concatenating their latent vectors
Learn flux conservation relationships solutions neighboring subdomains the encuded lasent space using the flux conservation autocnoodor
CoAE-MLSim
Figure 39: Flow chart for training the CoAE-ML approach
34


Reviews:
[{"review_id": "lnoE4VUAXUM", "reviewer": "Reviewer_dyhE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the clarity of the paper needs some seriour improvement. \n\nQuestions/Weaknesses:\n1. The related work sections covers a wide range of works that related to pde solving using neural networks. Although the paper provided a great list of related works, the relationship between this work and those mentioned is unclear to me. This is a critical piece that I am missing. If the related work section is delivered with more emphasis on the relationship instead of a mere list, it would be much improved.\n\n2. The clarity of method description needs some significant improvement. Possibly rising from the size of the figures being too small, I had a hard time understanding the general flow and what to expect for the method proposed. A lot of questions like what is the dimension of each latent space, what is the size of your encoder network and do all the subdomains share the same encoder (I believe so)? How does the solution conditional AE take the source and geometry as conditions? Are they just appended to the input of both encoder and decoder?  \n\n3. The autoencoders usually does not generalize well for the unseen data and the decoded result from unseen latent space is usually not valid in the original space unless being regularized properly (like Variational autoencoder etc.). How does this method do when facing out-of-distribution test cases?\n\nCouple of minor points:\n1. I believe the Figure 1 is not mentioned in the main text?\n2. The figures are really too smallto see clearly. It heavily undermines the clarity of the paper\n3. Page 4 alg 1 line 5 there is a error in latex? I believe it is <?\n4. What is \\THETA in page 4 alg 1 line 8? \n5. In Section 3.4 line 6 what is the citation [Ranade et al., 2021a] for? CoAE-MLSim is the proposed algorithm right?\n6. For result part, again the size of the plot makes it impossible to read\n7. The final table, there is no caption or table label and the unit is unclear", "summary_of_the_paper": "The paper propsed a composable AE-based PDE solver that combines traditional PDE solver with ML techniques. It split the big domain into sub-domains and compress the high dimensional representation into lower dimension using autoencoder.", "summary_of_the_review": "The paper is lacking in clarity as mentioned above. Although technically might be novel and can benefit the scientific computing community, this is far from publishable from a perspective of readability. \n\nUpdate on Nov. 20th during rebuttal:\nI think during the rebuttal period, I have gain much more understanding of the paper and willing to raise the score. My remaining concerns are clarity of the paper (including the fact that I still need to use 200% to see whats in the figures, but understandable given the page limit) and out-of-distribution generalizability.\n\nThe paper raised an interesting idea, using changing the simulation to the latent space by compression of autoencoders. However as the overall goal is still to simulate new designs, the generalizability of it due to autoencoders might still bottleneck its practicle impact.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "flQZ2PuOoYu", "reviewer": "Reviewer_ji7S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:* \n- I think this is an important application area with growing interest. \n- The approach seems smart to me. In particular, I could imagine this parallelizing well, it makes sense to me that applying the same network to each subdomain would allow for simpler networks, and it makes sense to me that learning low-dimensional representations with autoencoders could help. This is especially interesting to me in the context of learning a representation for the PDE conditions that you wish to generalize over. \n- My favorite part might be the dynamic inferencing with the iterative algorithm to establish consistency, enabling the networks to focus on subdomains. \n- Being able to initialize the ML solution with coarse-grained solutions from the PDE solver to speed up convergence is also intriguing. \n- I appreciate that the paper included results on quite a few problems, covering a variety of scenarios (steady-state vs. transient and generalizing across boundary conditions, sources, geometry, and initial conditions). \n\n*Weaknesses:*\n \nI think that:\n- The comparison to baselines could be improved. \n- Some of the claims are not carefully backed up. \n- The explanation of the relationship to the existing literature could be improved. \n\n*More details on the above weaknesses:*\n\n*Comparison to baselines:*\n\n\"We did not find good benchmarks to compare our unsupervised, iterative inferencing algorithm against\" I think this is a slightly unfair comment. The unsupervised and iterative inferencing aspects are only positives if they have the claimed benefits, as compared to other ML methods (more accurate and better generalization). There is a lot of recent work addressing the same ML task (as mentioned in the related work section.) This paper contains some comparisons to previous work, but as I detail below, there seem to be some holes.\n\nFCNN is by far the strongest competitor for the Laplace example in the appendix. Why is this left off of the baseline comparison table in the main paper? Further, is there any reason that FCNN couldn't have been used for the other examples? \n\nWhy is FNO not applied to the Chip cooling (Temperature) example?\n\nA major point in this paper is improved generalization across PDE conditions. However, I think that's hard to check when only looking at the test errors for each method. In other words, is CoAE-MLSim's error lower than UNet's error because the approach fit the training data better, or is it because it generalized better? Further, in some cases, it's not obvious to me if the test errors are impressive, so maybe it is having a hard time generalizing. It would be helpful to see train vs. test errors, and ideally I like to see train vs. val. vs. test. \n\nFor the second main example (vortex decay over time), looking at Figures 8 and 33 (four of the fifty test conditions), CoAE-MLSim has much lower error than the baselines in the extrapolation phase but noticeably higher in the interpolation phase. In some cases, it's hard to tell how close the FNO line is to zero - it could be that CoAE-MLSim even has orders of magnitude more error. Since we can see that there's a big difference between interpolation and extrapolation, it would be helpful to see the test error averaged over the 50 test cases but not averaged over the 50 time steps. When averaged over all 50 time steps for the table on page 9, it could be that CoAE-MLSim looks better than FNO just because of the extrapolation regime. In practice, someone might pick FNO over CoAE-MLSim if they aren't interested in extrapolating in time. Do the results in the table for vortex decay back up the claim that CoAE-MLSim is generalizing over initial conditions better than FNO, or is it just better at extrapolation in time?\n\n*Backing up claims:*\n\nThe abstract says that the method is tested for a variety of cases to demonstrate a list of things, including \"scalability.\" The list of \"significant contributions\" also includes \"This enables scaling to arbitrary PDE conditions...\" I might have missed/forgotten something, but I think this wasn't tested?\n\n\"Hence, the choice of subdomain size depends on the trade-off between speed and accuracy.\" This isn't clear to me from the results. It seems like 32^3 is the fastest and most accurate? \n\nI noticed some other claims that I think are speculations, not backed up with reported experiments. If I didn't miss something, this could be fixed by adding words like \"might.\" \n- \"Physics constrained optimization at inference time can be used to improve convergence robustness and fidelity with physics.\"\n- \"The decoupling allows for better modeling of long range time dynamics and results in improved stability and generalizability.\"\n- \"Each solution variable can be trained using a different autoencoder to improve accuracy.\"\n- \"Since, the PDE solutions are dependent and unique to PDE conditions, establishing this explicit dependency in the autoencoder improves robustness.\"\n- \"Additionally, the CoAE-MLSim apprach solves the PDE solution in the latent space, and hence, the idea of conditioning at the bottleneck layer improves solution predictions near geometry and boundaries, especially when the solution latent vector prediction has minor deviations.\"\n- \"It may be observed that the FCNN performs better than both UNet and FNO and this points to an important aspect about representation of PDE conditions and its impact on accuracy.\" The representation of the PDE conditions could be why, but it's hard to say without careful ablation studies. There's a lot different about the networks. \n- Similarly: \"Furthermore, compressed representations of sparse, high-dimensional PDE conditions improves generalizability.\"\n\n\n*Relationship to literature:*\n\nThe citation in this sentence is abrupt and confusing because it sounds like CoAE-MLSim is a method from that paper instead of the new method: \"Figure 4 shows a schematic of the autoencoder setup used in the CoAE-MLSim (Ranade et al., 2021a).\" More broadly, Ranade et al., 2021a, Ranade et al., 2021b, and Maleki, et al., 2021 are all cited and all quite related to this paper. It should be more clear how the authors are building on those papers (what exactly they are citing them for), and which parts of CoAE-MLSim are new. (The Maleki part is clearer in the appendix, but the reader shouldn't have to check the appendix to know what is new in a paper.) \n\nI thought that otherwise the related work section was okay but was largely just summarizing some papers without giving context for how they relate to this paper. \n\n\n~~~~\n\n\n\nAdditional feedback (minor details, could fix in a later version, but no need to discuss in the discussion phase):\n- The abstract could be clearer about what the machine learning task is that CoAE-MLSim addresses. \n- The text in the figures is often too small. \n- \"using pre-trained decoders (g)\" - probably meant g_u? \n- Many of the figures would be more clear if they said pre-trained solution encoders & solution decoders, since there are multiple types of autoencoders.\n- The notation is inconsistent, especially with nu. For example, the notation in Figures 2 & 3 doesn't seem to match the notation in Alg 1. Then on Page 4 & Figure 4, the notation changes again.\n- Why is the error table not ordered 8^3, 16^3, 32^3 like Figure 9? The order makes it harder for the reader to reason about the tradeoff. \n- Why is Err(T_max) negative sometimes? Maybe I don't understand the definition, but I would expect to use absolute value?\n- I don't think the study about different subdomain sizes is an \"ablation\" study since they aren't removing a component of the method. \n- Figure 11: I'm guessing that the y-axis is log error, but this isn't labeled as such. I didn't really understand the the legend or the figure in general until I got to the appendix, since there's little discussion of it in the main paper. \n- \"Figure 30 shows comparisons of CoAE-MLSim with Ansys Fluent for 4 unseen objects in addition to the example shown in the main paper.\" - probably from previous draft. Now this whole example is in the appendix, unless I missed something. \n- My understanding is that each type of autoencoder is trained separately and that there's an ordering that makes sense to do this in, so you can use one trained autoencoder for the next one (i.e. train the PDE condition AEs, then the PDE solution AE, then the flux conservation AE, then the time integration AE). This took me a while to understand though, so maybe this could be mentioned in the body of the paper. (Or perhaps I missed that!)\n- It seems that the time integration autoencoder isn't actually an autoencoder if it's outputting the solution at the next time step, not reconstructing the input. \n- Either I don't understand Figure 5 or the labels are wrong. \n- It's implied in the paper (like in Algorithm 1) that the boundary conditions are encoded like the other PDE conditions. In the Appendix (A.1), it's stated that \"The training portion of the CoAE-MLSim approach proposed in this work corresponds to training of several autoencoders to learn the representations of PDE solutions, conditions, such as geometry, boundary conditions and PDE source terms as well as flux conservation and time integration.\" But then later in the appendix (A.1.3), it's stated that boundary conditions could be learned with autoencoders but are actually manually encoded for this paper. That seems misleading. ", "summary_of_the_paper": "This paper proposes a new ML approach called CoAE-MLSim that is a faster alternative to PDE solvers. Compared to previous ML work on this problem, it aims to be more accurate and generalize better across PDE conditions. They also aim to require fewer PDE solutions to train the model. ", "summary_of_the_review": "I think that CoAE-MLSim seems like a good idea and could be a great method. However, the results are not yet strongly backing up the claims, especially that it is more accurate and generalizes better than the previous ML methods for this task. I think that a revision during the discussion phase could raise my score to 8/accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XJuQ33BjmYc", "reviewer": "Reviewer_hE8v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method proposed in this paper is novel and validated. Since this work is about accelerating the PDE algorithms, I would expect to read more discussions on how or why this method can be faster than traditional solvers by the ratio given by the authors (e.g., 40-50x faster). Besides, it would be more solid if the author compare between using different sizes of layers (number of parameters) in the autoencoders. How is the balance between performance, accuracy, as well as generalization ability, regarding to the size of the autoencoders? How to choose the layer size regarding to different problems or some performance requirement?", "summary_of_the_paper": "This paper used autoencoders to approximate different phases of a PDE solver.", "summary_of_the_review": "I would like to see the acceptance of this paper if the authors can provide more comparisons and discussion on the performance / accuracy regarding to the size of layers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VuaNPH7Aq72", "reviewer": "Reviewer_WDpc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method uses the idea of domain decomposition, and thus it has the potential to only need a small dataset to train the networks.\n\nWeaknesses:\n- The method requires 100-1000 PDE solutions for training, which is not really a very small dataset as they claimed. For a comparison, in the paper arXiv:2010.08895, it also uses 1000 solutions for the NS equation, and in this paper, it uses 500 in Section 4.2.\n- The method part is not well written and hard to read. After reading Section 3.2/3.3, I only capture the main idea, but the detail of the algorithm is very vague and not well presented, e.g., how the neural networks are used here, what the network input and output are, how the flux conservation is applied, etc. The authors should rearrange Section 3 so that it is easy to read.\n- During the inference stage, because the method needs to couple all the solutions in subdomains together by using an iterative algorithm. The inference time is significantly slower than other ML methods, which only requires one forward pass of the network.\n- Also, it is not fair to compare the speed with Ansys solver, because Ansys is much more accuracy than the proposed method. If we need to compare with Ansys solver, then the authors should use a low resolution mesh of Ansys solver, such that it has a low accuracy similar to CoAE-MLSim, and then we can compare the speed.\n- The accuracy is not very good. In Section 4.1, the MSE of the temperature is 5.3e-2, so without the square, the error is > 0.2, which is really large.\n- For the result in Section 4.2, the timestep is dt = 1e-4, and here the authors only predict for 50 timesteps, i.e., t = 50 * dt = 0.005, which is too small. Even for the first few timesteps, the L2 error in Fig. 8 is almost 0.1, which is large.\n- The authors claims that the method has a good generalization, but there is no evidence to support this. The prediction of future time in Fig. 8 is not really extrapolation. For example, we would not say time series prediction is extrapolation, and it is still interpolation in the space of PDE condition/solution.\n\nOther comments:\n- What does “CoAE-MLSim” stands for? Composable Autoencoder - Machine Learning Simulation? In the whole paper, “Composable” is only mentioned in the title, but not any where in the text, and thus it is not clear what it means exactly.\n- In the abstract, the authors only mentioned the name of the method. After reading the abstract, I have no idea what “CoAE-MLSim” is.\n- In the abstract, “unsupervised” is conflict with “relatively fewer samples of PDE solutions”. The method is not unsupervised.\n- The Point 4 “The iterative inferencing algorithm is completely unsupervised and allows for coupling with traditional PDE solvers.” The iterative algorithm is part of inference, and the inference of networks of course doesn’t need the data. It is strange to say the inference stage is “unsupervised”. Also in Section 3.8.\n- In Algorithm 1 Line 5, there is a “?”.\n- There is empty space in the top right corner of Page 6.\n- Is the SDF for the local domain or the global domain?\n- In Fig. 4, the SDF looks strange. SDF is a continuous function, but in the figure, it changes from red color to blue color sharply and then keep the same blue color.\n- The fonts in many figures are too small to read.\n", "summary_of_the_paper": "The paper proposed CoAE-MLSim to learn with relatively fewer samples of PDE solutions and solve PDEs. CoAE-MLSim uses the idea of domain decomposition: first learn the solution on local subdomains using autoencoder, and then couple these sub-solutions by an iterative algorithm. Numerical experiments are performed to test the efficiency of the method.", "summary_of_the_review": "The main issue is that their numerical experiments are not convincing enough to support their claims on small data, accuracy, and extrapolation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach called CoAE-MLSim as a faster alternative to PDE solvers.  According to the authors, the main strength of the method is that it needs fewer training data, however, as pointed out by the reviewers, the factor is not significant enough in the current results. Another concern raised by the reviewers is that the iterative algorithm is much more expensive than only one forward pass of other ML methods. Furthermore, the reviewers criticized that the comparisons with baselines are not sufficient, and some claims in the paper were not backed up. The authors provided their rebuttals and had a long discussion with most of the reviewers. Some clarifications have been made, and some reviewers increased their scores accordingly. However, overall speaking, the major problems with the paper still remain, and the rebuttal has not successfully convinced the reviewers to turn to the positive side. Therefore, we cannot give a green light to this paper yet.
Decision: Reject