Composing Partial Differential Equations with Physics-Aware Neural Networks
Authors: Matthias Karlbauer
Timothy Praditia
Sebastian Otte
Sergey Oladyshkin
Wolfgang Nowak
Martin V. Butz
Venue: ICLR 2022 Submitted (2022)
Keywords: physics-aware neural networks
partial differential equations
advection-diffusion equations
learning constituents
out-of-distribution generalization

Abstract: We introduce a compositional physics-aware neural network (FINN) for learning spatiotemporal advection-diffusion processes. FINN implements a new way of combining the learning abilities of artificial neural networks with physical and structural knowledge from numerical simulation by modeling the constituents of partial differential equations (PDEs) in a compositional manner. Results on both one- and two-dimensional PDEs (Burger's, diffusion-sorption, diffusion-reaction) demonstrate FINN's superior process modeling accuracy and excellent out-of-distribution generalization ability beyond initial and boundary conditions. With only one tenth of the number of parameters on average, FINN outperforms pure machine learning and other state-of-the-art physics-aware models in all cases---often even by multiple orders of magnitude. Moreover, FINN outperforms a calibrated physical model when approximating sparse real-world data in a diffusion-sorption scenario, confirming its generalization abilities and showing explanatory potential by revealing the unknown retardation factor of the observed process.

Under review as a conference paper at ICLR 2022
COMPOSING PARTIAL DIFFERENTIAL EQUATIONS WITH PHYSICS-AWARE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a compositional physics-aware neural network (FINN) for learning spatiotemporal Divection-diffusion processes. FINN implements a new way of combining the learning abilities of artificial neural networks with physical and structural knowledge from numerical simulation by modeling the constituents partial differential equations (PDEs) in a compositional manner. Results both one- and two-dimensional PDEs (Burger's, diffusion-sorption, diffusion-reaaciinn, Allen-Cahn) demonstrate FINN's superior modeling accuracy and excellent out- of-distribution generalization ability beyond initial and boundary conditions. With only one tenth of the number of parameters on average. FINN outperforms pure machine learning and other statc-of-the-an physics-aware models in all cases- often even by multiple orders of magnitude. Moreover, FINN outperforms cali- brated physical model when approximating sparse real-world data in diffusion- sorption scenario, confirming its generalization abilities and showing explanatory potential by revealing the unknown retardation factor of the observed process.
INTRODUCTION
Artificial neural networks (ANNs) considered universal function approximators (Cybenko 1989). Their effective learning ability, however, greatly depends on domain and task-specific pre- structuring and methodological modifications referred to as inductive biases (Battaglia al. 2018). Typically, inductive biases limit the space of possible models by reducing the opportunities for com-- putational shortcuts, which can lead erroneous implications derived from potentially limited dataset (overfitting). The recently evolving field of physics-informet machine learning employs physical knowledge as inductive bias providing huge generalization advantages in contrast to pure machine learning (ML) in physical domains (Raissi et al. 2019). While numerous approaches have been introduced to augment ANNs with physical knowledge, these methods either do not allow the incorporation of explicitly defined physical equations (Long et al. 2018 Sco et al. 2019: Guen & Thome, 2020 Li ct al. 2020a, Sitzmann et al. 2020) or cannot generalize to other initial and boundary conditions than those encountered during training (Raissi et al. 2019).
In this work, we present the finite volume neural network (FINN) model physics-an physics-aware aware neural network structure adhering to the idea of spatial and temporal discretization in numerical simulation. FINN consists of multiple neural network modules that interact in a distributed, compositional man- (Battaglia et al. 2018 Lake al. 2017: Lake 2019). The modules designed to account for specific parts of advection diffuusin equations, a class of partial differential equations (PDEs). This modularization allows combine two advantages that are not yet met by state-of-the-ant models: the explicit incorporation of physical knowledge and the generalization over initial and boundary conditions. To the best of our knowledge, FINN's ability to adjust to different initial and boundary conditions and to explicitly learn constitutive relationships and reaction terms is unique, yielding excellent out-of-distribution generalization. The core contributions of this work are:
Introduction of FINN, a physics-awar neural network model, explicitly designed to gener- alize over initial and boundary conditions, demonstrating excellent generalization ability.
Evaluation of state-of-the-a pure ML and physics-aware models, contrasted to FINN on one- and two-dimensional benchmarks, demonstrating benefits of explicit model design.
Application of FINN to a worldworld contamination -ifffaationniffusio problem, verifying its appli- cability to real, spatially and temporally constrained training data.

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Non-physics-aware ANN architectures Pure MI. models that are designed for spatiotemporal data processing can be separated into temporal convolution (TCN, KKacchrenner et al., 2016) and recurrent neural networks. While the former perform convolutions over space and time, representa- tives of the latter, e.g., convolutional LSTM (ConyLSTM. [hhi et al. 2015) or DISTANA (Karlbauer et al. 2019). aggregate spatial neighbor information to further process the temporal component with recurrent units. Since pure ML models do not adhere to physical principles, they require enormous amounts of training data and parameters in order to approximate desired physical process; but still not guaranteed to behave consistently outside the regime of the training data.
Physics-aware ANN architectures When designed to satisfy physical equations, ANNs are re- ported to have greater robustness in terms of physical plausibility. For example, the physics- informed neural network (PINN, Raissi et al., 2019) consists of an MLP that satisfies an explicitly defined PDE with specific initial and boundary conditions, using automatic differentiation. How- ever, the remarkable results beyond the time steps encountered during training are limited to the very particular PDE and its conditions. A trained PINN can actually not applied to other different initial and boundary conditions, which limits its applicability in real-world scenarios.
Other methods by Long et al. (PDENet, 2018), Guen & Thome (PhyDNet, 2020), or Sitzmann et al. (SIREN, (020) learn the first n derivatives to achieve a physically plausible behavior, but lack the option to icllude physical equations. The same limitation applies when operators are learned to ap- proximate PDEs (Li et al. (D2 or when physics-aware graph neural networks are applied (Seo) et al.) 2019). Yin et al. (APHYNITY, 2020) propose to approximate equations with an appropriate physical model and to augment the result by an ANN, preventing the ANN to approximate distinct part within the physical model. For more comparison to related work, please refer to subsectiion A.I
In summary, none of the above methods can explicitly learn particular constitutive relationships or reaction terms while simultaneously generalizing beyond different initial and boundary conditions.
FINITE VOLUME NEURAL NETWORK (FINN)
Problem formulation Here, we focus modeling spatiotemporal physical processes. Specifi- cally, we consider systems governed by advection
dddeecion-diifusionn diffusion type equations (Smolarkiewicz) 1983):
Ou Øt D(u) dz v(u)
82m
Ou Ōr + q(u),
where is the quantity of interest, is time, is the spatial coordinate, D is the diffusion coeffi- cient, v is the advection velocity, and y is the source/sink term. [Eq. caan be partitioned into three parts: the storage term, the flux terms, and the source/sink term. The storage term 3t the change of the quantity u over time. The flux terms are the advective flux v(u) &u and the diffu- az 8g2 Both calculate the amount of exchanged between neighboring positions. The source/sink term (u) describes the generation or elimination of the quantity u. [Eq. 1 is a general form of PDE with up to second order spatial derivative, but it has wide range of applicability due to the flexibility of defining D(u), v(u), and g(u) as different functions of u, as is shown by the numerical experiments in this work.
Ou describes
sive flux (u) 0"
The finite volume method (FVM, Moukalled et al. 2016) discretizes simulation domain into con- trol volumes (i 1, ...... N₂), where exchange fluxes are calculated using a surface integral (Riley et al. 2009). In order to match this structure in FINN, we introduce two different kernels, which are (spatially) casted across the discretized control volumes: the flux kernel, modeling the flux terms, and the state kernel, modeling the source/sink term as well as the storage term. The overall FINN architecture shown in Figure 1
Flux kernel The flux kernel F approximates the surface integral for each control volume i with boundary 52 by a composition of multiple subkernels fj. each representing the flux through a dis- cretized surface element j:
No
of Swen
8u (D(u)
Ou v(u)
=
Σ f1
0x22
dz
f dl',
2

---

Under review as a conference paper at ICLR 2022
10
ODE sulxes
F++
VM--
V2
17 (M))
1 (16))
(1x)
P.N
F:
8.
Figure 1: Flux and state kernels in FINN for the mneddimensionll case (left); detailed assignment of the individual modules with their contribution to Eq.. T (right). Read lines indicate gradient flow.
where Ns, is the number of discrete surface elements of control volume is continuous surface element (a subset of 12). fy are subkernels (which are realized as feedforward network modules), and à is the unit normal vector pointing outwards of w. In our exemplary -iedimmesioonl arrangement, two subkernels fi- and Ji+ (see (Feere Tb contain the modules (pp, SAA. and yP (PN, or 9P; for Ji- and fi+, respectively). The module yP linear layer with the purpose learn the numerical FVM stencil. More specifically, the module P.N-- takes the inputs ut and its neighbor u;-1. while PP,,, takes the inputs u, and U++1, and output the approximation of the spatial derivative as This signifies that the weight of &axr should amount to [-1,1] with respect to [u4, u1=1] and [u;, u++1] (i.e. simple difference between neighboring control volumes) in ideal mieedinnnioonal problems.
The other modules A and yD are responsible for advective and diffusive fluxes, respectively. Both modules receive only u; as input. Moreover, the module R applies an upwind differencing scheme, which prevents numerical instability in the first order spatial derivative calculation (Versteeg & Malalasekera 1995). ensures that calculation of the advective fluxes using 4 performed only on one control volume surface (either left or right, and not both at the same time), whereas calcula- tion of the diffusive fluxes using yp is performed on both surfaces. This means, for the advective flux calculation, the summation of the numerical stencil from both fi- and fi+ will only lead to [-,, which corresponds to either [4s. 14-1] when PA > 0 or [u,, M4+1] when PP < O (i.e. only first order spatial derivative). For the diffusive flux calculation, the summation of the numerical stencil will lead to the classical one-dimensional numerical Laplacian with [1, 22 1] corresponding to (u=1, uy 1,+1], that is, the second order spatial derivative, because the calculation of yD is per- formed on both surfaces (see (subsection A.3 for more details). Accordingly, module R generates the inductive bias to make GA only approximate the advective, and pp the diffusive flux.
For the advective flux calculation, if the velocity v is function of the module 9.4 approximates this dependence using feedforward neural network, i.e. ( v(u). Otherwise, 4 scalar value A v, which can be also set as learnable parameter. The output of 4 is activated by the following case-sensitive ReLU R:
R(ea) = (Re.UU(Pa), 1 -RRLLU(-pa) on fits
on fi=:
(3)
and then multiplied with the output of you (i.c. 22) to obtain the corresponding advective flux at the surface. Similarly for the diffusive fluxes, if the diffusion coefficient D depends on u. the module pp takes Otherwise, pp is a scalar value PD = D, which can also be set as a learnable parameter. The output of yp is not activated by the module R. but is directly multiplied with the output of ya to obtain the corresponding diffusive flux at the surface. Consequently, integration of subkernels ʃ3 in Eq. and substitution of Eq. leads the following representation:
input to learn the function (u)
D(u) using feedforward neural network.
F. [o 44- 44 P.l pN u,+ IPplo if 0, (u; ")-1) + pur u441)(pp - PAlt otherwise.
(4)
Boundary conditions A means of applying boundary conditions in the model is essential when solving PDEs. Currently available models mostly adopt a convolutional structure for modelling spatiotemporal processes. However, convolutional structure only allows a constant value to be
3

---

Under review as a conference paper at ICLR 2022
padded at the domain boundaries (e.g. zero-padding mirror-padding), which is only appropriate for the implementation of Dirichlet or periodic boundary condition types. Other types of frequently used boundary conditions are Neumann and Cauchy. They are defined as a derivative of the quantity of interest, and hence cannot be easily implemented in convolutional models. With certain pre- adjusted accordingly to
defined boundary condition types, the flux kernels at the boundaries allow for straightfrwwar boundary condition implementation. For Dirichlet boundary condition, a constant value
Up is set as the input u (for the flux kernel fi-) or ui+1 (for fi+) at the corresponding boundary. For Neumann boundary condition v, the output of the flux kernel fi- or fi+ at the corresponding boundary is set to be equal to v. With Cauchy boundary condition, the solution-dependent derivative is calculated and set as Wi-1 or 1 at the corresponding boundary.
State kernel
The state kernel S calculates the source/sink and storage terms of Eq.. 1 source/sink (if required) is learned using the module Dy ~ q(u), which takes u as input. The 0u is then calculated using the output of the flux kernel and module Py of the state kernel: storage, 0e
The
Ou: Öt
S: = +v R
By doing so, the PDE in Eq. iss now reduced to system of coupled ordinary differential equations (ODEs), which are functions of u; Mi-1, 1, and l. Thus, the solutions of the coupled ODE system can be found using numerical integration over time. Since first order explicit approaches, such as the Euler method (Butcher (208)) suffer from numerical instability (Courant et al. 1967; Tsaacson & Keller| 1994). we employ the neural ordinary differential equation method (Neural ODE, Chen itt 2018)) to reduce numerical instability via the Runge-Kutta adaptive time-stepping strategy. When using the Neural ODE method, the predicted values of u at time are fed back into the network as direct inputs to predict u at time & + 1. Therefore, the entire training is performed in closed-loop, improving stability and accuracy of the prediction compared to networks trained with teacher forcing, i.e. one-step-ahead prediction (Praditia et al. 2020). The weight update is realized by applying backpropagation through time (indicated by red arrows in Figure 1). In short, FINN takes only the initial condition at time t = 0 and propagates the dynamics forward.
4 EXPERIMENTS, RESULTS & DISCUSSION
4.1 SYNTHETIC DATASET
To demonstrate FINN's performance in comparison to other models, four different equations are considered as applications. First, Burger's 's equation (Basdevant et al. 1986) is chosen as a chal- lenging function, as a non-linear PDE with v(u) tion u(I, t). Second, the diffasion-sorption equation (Nowak & Guthke, 2016) is selected with the non-linear retardation factor R(u) as coefficient for the storage term, which contains a singularity R(u) + xc for + 0 due to the parameter choice. Third, the two- awo-dimensional equation (Klaasen & Troy 1984) as candidate for a diffusion--eaacn reaction equation (Turing) 1952) is selected which is challenging because it consists of two non-linearly coupled PDEs solve two main unknowns: the activator u1 and the inhibitor U2. Fourth, the Allen-Cahn equation with a cu- bic reaction term was chosen, leading multiple jumps in the solution u(z, t). Details on all four equations, data generation and architecture designs can be found in suusecction A.4| |uuseection A.5) subsection A.6 subsection A.7 of the appendix, respectively.
that could lead to a shock in the solu-
Brrchugh-aguume
For each problem, three different datasets are generated (by conventional numerical simulation): train, used train the models, in-distribution test (in-dis-test), being the train data simulated with a longer time span to test the models' generalization ability (extrapolation), and out-of-distriben test (out-dis-text). Out-dis-test data are used to test a trained ML model under conditions that are far away from training conditions, not only in terms of querying outputs for unseen inputs. Instead, out- dis-test data query outputs with regards to changes not captured by the inputs. These are changes that the ML tool per definition cannot be made aware of during training. In this work, they are represented by data generated with different initial or boundary condition, to test the generalization ability of the models outside the training distributions. FINN is trained and compared with both spatiotemporal deep learning models such as TCN, ConvLSTM, DISTANA and physics-award neu- ral network models such PINN and PhytNet. All models are trained with ten different random
4

---

Under review as a conference paper at ICLR 2022
Table Comparison of MSE and according standard deviation scores across ten repetitions between different deep learning (above dashed line) and physics-aware neural network (below dashed line) methods on the different equations. Best results are reported in bold.
Dataset
Eqn.
Model
Params
Truin
In-dis-test
Out-dis-test
TCN
38 500
(1.6 10
(1.8 3.1) 10"1
(1.6+ 10
COmVLSTM
132000
(6.8 x 10
(1.2 10
(7.3 10
Burger
DISTANA
126
(1.8 x 10
(4.0 x 10-3
(..5 x
------
PINN
3021
(5.1 10
(5.011 10-3
PhyDNet
37718
(7.2 10
(1.8 10-1
(4.5 x 10
421
(2.8+ 10-6
(2.5+3.1) 10-6
(2.8 2.9) 10-6
TCN
3834
(9.7+ x 10-2
(1.2 x 10
(1.1 10-1
Diffusion- sorption
COM
3960
(3.2+29)) x 10-
(3.0+23) x
(5.8+ x 10-2
DISTANA
3739
(4.6+25) x
x.4+2))x10 10
x
PINN
3042
(4.7+84) x 10-5
(4.1 10-
PhyDNet
37815
(3.5 10-5
(9.1+15.4) 10-3
(1.7 0.9) 10-2
FINN
528
(4.7 x x
(1.3113) x 10
(4.1 4.0) x 10 -
TCN
31 734
(1.4+0.9) x 10-
(4.7+ x 10-1
(1.5+0.8) x 10
Ditfusion-
ComVLSTM
24440
(8.7 x 10-3
(9.3 10
(1.5 10
reaction
DISTANA
75.29
(4.0 x 10
(1.8 10
(1.3 x 10-
3062
(2.7118) 10-4
(7.0 10
PhyDNet
185 589
(7.5 (..5).9)) x 10
(7.8118) x 10
(3.5 13) 10
FINN
882
(1.3+ x 10
(2.1 x 10
(6.1 = 0.3) x 10
seeds using PyTorch's 's default weight initialization, and mean and standard deviation of the predic- tion errors are summarized in Table for train, in-dis-test and out-dis-test, The details of each run reported in the appendix. It is noteworthy that PINN cannot be tested the out-dis-test dataset, since PINN assumes that the unknown variable is an explicit function of x and f, and hence, when the initial or boundary condition is changed, the function will also be different and no longer valid.
Burger's The predictions of the best trained model of each method for the in-dis-test and out- dis-test data are shown in Figure and Figure 3] respectively. Both TCN and ConvLSTM fail to produce reasonable predictions, but qualitatively the other models manage to capture the shape of the data sufficiently, even towards the end of the in-dis-test period, where closed loop prediction has been applied for 380 time steps (after 20 steps of teacher forcing, see (subsection A.4 for details). DISTANA, PINN, and FINN stand out in particular, but FINN produces more consistent and curate predictions, evidenced by the mean value of the prediction errors. When tested against data generated with different initial condition (out-dis-test), all models except for TCN and PhyDNet
Dota
TCN
ConVLSTM
DISTANA
PINN
FhyDNet
FINN
1
0
0
0
0
-1
-1
-1
-1
-1
-1
2
2
2
0.5
0.5
0.5
0.5
0.5
0.5
0.5
-0.5
0.5
4.5
0.5
005
-0.5
-0.5
0
0
0
1
-1
-1
Figure 2: Plots of Burger's data (red) and in-dis-test (blue) prediction using different models. The plots in the first row show the solution over æ and f (the red lines mark the transition from train to in-dis-test), the second row visualizes the best model's solution distributed in at 2.
5

---

Under review as a conference paper at ICLR 2022
Data
TCN
ConvLSTM
DISTANA
PhyDNet
FINN
0.5
X
0
0
0
0
-1
-1
0
0.5
1
0
0.5
0.5
0.5
1
0
0.5
1
0
0.5
t=1 1
t=1
1
1
t= 1
t=
0.5
0.5
0.5
0.5
0.5
0.5
0
-0.5
0.55
-05
-0.5
0.5
00.5
0
-1
0
0
1
Figure 3: Plots of Burger's data (red) and prediction (blue) of out-dis-test data using different mod- els. The plots in the first row show the solution over x and t, the second row visualizes the best model's solution over x at
perform well. However, FINN still outperforms the other models with significantly lower predic- tion error. The advective velocity learned by FINN's module şA is shown in Figure 8) (left) and verifies that it successfully learned the advective velocity be described by an identity function.
Diffusion-sorption The predictions of the best trained model of each method for the concentration from the diffusion-sorption equation are shown in Figure 4]and Figure TCN and Conv are shown to perform poorly even the train data, evidenced by the high mean value of the prediction errors. On in-dis-test data, all models successfully produce relatively accurate predictions. How- ever, when tested against different boundary conditions (out-dis-test), only FINN is able to capture the modifications and generalize well. The other models are shown to still overfit to the different boundary condition used in the train data (as detailed in subsection A.5). The retardation factor R(u) learned by FINN's ypD module is shown in Figure 8 (second from left). The plot shows that the module learned the Freundlich retardation factor with reasonable accuracy.
The predictions of the best trained model of each method for activator un in the diffusion-reeacionn equation are shown in Figure 6 and Figure 7, TCN is again shown to fail to learn sufficiently from the train data. On in-dis-test data, DISTANA and the physics-aware models all predict with reasonable accuracy. When tested against data with different initial condition (out- dis-test), however, DISTANA and PhyDNet produce predictions with lower accuracy, and we find that FINN is the only model producing relatively low prediction errors. The reaction functions learned by FINN's s P module are shown in [Figure The plots show that the module successfully learned the Fitzhugh-Nagumo reaction function, both for the activator and inhibitor.
Diffusion-reaction
Discussion Even with high number of parameters, the prediction errors obtained using the pure ML methods (TCN, ConvLSTM, DISTANA) are worse compared to the physics-aware models, as
Dota
TCN
ConVLSTM
DISTANA
PINN
PhyDNct
FINN
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.8
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.4
0.0
0.0
10000
0.0
10000
0.0
10000
0.0
10000
0.0
10000
0.0
10000
0.0
10000
10000
10000
10000
10000
10000
10000
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0
0
0
0.5
1
0.5
0.5
0
0.5
1
0.5
0.5
0.5
Figure 4: Plots of diffusion-sorptionn's dissolved concentration (red) and in-dis-test prediction (bluc) using different models. The first row shows the solution over z and (red lines mark the transition from train to in-dis-test), the second row visualizes the solution distributed in at t 10000.

---

Under review as a conference paper at ICLR 2022
Data
TCN
ConvLSTM
DISTANA
PhyDNet
FINN
1.0
1.0
1.0
1.0
1.0
0.6
x
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
10000
0
10000
10000
0
10000
10000
10000
= 10000
t= 10000
10000
== 10000
t 10000
= 10000
0.5
0.5
0.5
0.5
0.5
0.5
0
0
0
0
0.5
0.5
0.5
1
0.5
0.5
0
0.5
Figure 5: Plots of diffusiinnssoptioons 's dissolved concentration data (red) and prediction (blue) of out-dis-test data using different models. The plots in the first row show the solution over x and t, the second row visualizes the solution distributed in æ at 10000.
shown in Table |] As a physics-aware model, Phy also possesses high number of parameters. However, most of the parameters are allocated to the data-driven part (i.e. ConvLSTM branch), compared to the physics-aware part (i.e. PhyCell branch). contrast to the other pure ML methods, DISTANA predicts with higher accuracy. This could act as an evidence that appropriate structural design of a neural network as essential as physical knowledge to regularize the model training.
Among the physics- models, PINN and PhyDNet lie on different extremes. On one side, PINN requires complete knowledge of the modelled system in form of the equation. This means that all the functions, such the advective velocity in the Burger's equation, the retardation factor in the diffusion-sorption equation, and the reaction function in the diffusion-reaction equation have to be pre-defined in order train the model. This leaves less room for learning from data and could be eror-prone if the designer's s assumption is incorrect. On the other side, PhyDNet relies more heavily on the data-driven part and, therefore, could overfit the train data. This can be shown by the fact that PhyDNet reaches the lowest training errors for the diffusion-sorption and diffusion-reactionn equa- tion predictions compared the other models, but its performance significantly deteriorates when applied to in- and out-dis-test data. Our proposed model, FINN, lies somewhere in the middle, com- promising between putting physical knowledge into the model and leaving room for learning from data. As consequence, FINN shows excellent generalization ability. It significantly outperforms the other models up to multiple orders of magnitude, especially on out-dis-test data, when tested with different initial and boundary conditions; which is considered particularly challenging task for ML models. Furthermore, the structure of FINN allows the extractions of learned functions such as the advective velocity, retardation factor, and reaction functions, showing good interpretability of the model.
We also show that FINN properly handles the provided numerical boundary condition, as evidenced when applied to the test data that generated with a different left boundary condition value, visual-
Data
TCN
ConVLSTM
DISTANA
PINN
FhyDNet
FINN
0.5
H
0
O
0
0
0.0
-1
-1
-1
-1
0.5
X
50.
50.
50,
50, y
50, Y
50, y
50, V
0.5
0.5
0.5
0.5
0.5
0.5
0.5
5'
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0
1
0
0
1
1
1
X
Figure 6: Plots of diffusion-reaction reaction's 's activator data uj (red) and extrapolated prediction (blue) using different models. The plots in the first row show the solution distributed over x and y at t = 50, and the plots in the second row show the solution distributed in x: at y and t = 50.
7

---

Under review as a conference paper at ICLR 2022
Data
TCN
ConVLSTM
DISTANA
PhyDNet
FINN
0.2
0.0
-0.2
1
1
0
0.4
= 10, y
0,4
= 10, y
=
0.4
t
10, y
10, y
0.4
10, y
0.4
t
10, y =
0
M
-0.4
0.4
0.4
-0.4
-0.4
0.4
1
Figure 7: Plots of diffussionrrecctoon activator data (red) and prediction (blue) of unseen dataset using different models. The plots in the first row show the solution distributed over x and y at t 10, and the plots in the second row show the solution distributed in y and / 10.
Data
Prediction
Data
Prediction
4.5
0.5
0.5
3.75
S' 0
S
0
0.0
$
0
0.0
-0.5
0.5
-1
1
0 UL
0
1
0
41
UL
UL
Figure 8: Plots of the learned functions (blue) as a function of u compared to the data (red) for Burger's (left) and diffusion-sorption (second from left). The learned activator in and inhibitor reaction functions (right plots of the pairs in center and right) in the diffusion-reaction equation are contrasted to the corresponding ground truth (left plots of the pairs).
ized in Figure 5
Here, the test data is generated with a Dirichlet boundary condition u(O, t) = 0.7, which is lower than the value used in the train data, w(0, t) = 1.0. However, FINN is the only model that appropriately processes this different boundary condition value so that the prediction fits the test data nicely. The other models overestimate their prediction by consistently showing tendency to still the prediction to the higher boundary condition value encountered during training.
Even though the spatial resolution used for the synthetic data generation is relatively coarse, leading to sparse data availability, FINN generalizes well. PINN, on the other hand, slightly suffers from the low resolution of the train data, although still shows reasonable performance for the three test cases. Nevertheless, we conducted an experiment showing that PINN performs slightly better and more consistently when trained higher resolution data (see appendix, [subsection A.10), albeit still worse than FINN on coarse data. Therefore, we conclude that FINN is also applicable to real observation data that are often available only in low resolution, and/or in limited quantity. We demonstrate this further in the next section, when we apply FINN to real experimental data. FINN's generalization ability is superior to PINN, due the fact that not possible apply a trained PINN model to predict data with different initial or boundary condition.
In terms of interpretability, FINN allows the extraction of functions learned by its dedicated modules. These learned functions can be compared with the real functions that generated the data (at least in the synthetic data case); examples are shown in Figure 8 The learned functions are the main data- driven discovery part of FINN and can also be used physical plausibility check. PhyDNet also comprises of a physics-aware and a data-driven part. However, it is difficult, if not impossible, to infer the learned equation from the model. Furthermore, the data-driven part of PhyDNet does not possess comparable interpretability. and could lead to overfitting as discussed earlier.
4.2 EXPERIMENTAL DATASET
Real observation data are often available in limited amount, and they only provide partial informa- tion about the system of interest. To demonstrate how FINN cope with real observation data, we use experimental data for the diffusion-sorption problem. The experimental data are collected from three different core samples that taken from the same geographical area: #1 #2, and #2B
8

---

Under review as a conference paper at ICLR 2022
Core #2
Core #1
Core #28
Retardation factor
600
5.25
400
R(u)
4.5
200
3.75
0
10 20
30
40
0
10
20
30
40
0.05
0.1
1000
2000
X
Figure 9: Breakthrough curve prediction of FINN (blue line) during training using data from core sample #2 (left), during testing using data from core sample #1 (second from left) and total concen- tration profile prediction using data from core sample #2B (second from right). The predictions are compared with the experimental data (red circles) and the results obtained using the physical model (orange dashed line). The learned retardation factor R(u) is shown in the right-most plot.
(see ssuseection A.12 in the appendix for details). The objective of this experiment is to learn the retardation factor function from one of the core samples that concurrently applies to all the other samples. For this particular purpose, we only implement FINN since the other models have no means of learning the retardation factor explicitly. Here, we use the module yD to learn the retarda- tion factor function, and assume that the diffusion coefficient values of all the samples known. FINN is trained with the breakthrough curve of u, which is the dissolved concentration only I loststend (i.e. only 55 data points).
FINN reaches higher accuracy for the training with core sample #2, with MSE = 4.84 x 10-4 compared a calibrated physical model from Nowak & Guthke (2016) with MSE = 1.06 x 10-3 because the latter has to assume a specific function R(()) with a few parameters for calibration. Our learned retardation factor is then applied and tested core samples #1 and #2B. Figure shows that FINN's prediction accuracy is competitive compared to the calibrated physical model. For core sample #1, FINN's prediction has an accuracy of 1.37 x 10-3 compared to the physical model that underestimates the breakthough curve (i.e. concentration profile) with MSE = 2.50 x 10-3 Core sample #2B has significantly longer length than the other samples, and therefore a no-flow Neumann boundary condition was assumed at the bottom of the core. Because there is break- through curve data available for this specific sample, we compare the prediction against the so-called total concentration profile u(z,fend) fend) at the end of the experiment. FINN produced prediction with an accuracy of 1.16 x 10 10-3, whereas the physical model overestimates the concentration with MSE 2.73 10-3 To briefly summarize, FINN is able to learn the retardation factor from sparse experimental dataset and apply it to other core samples with similar soil properties with rea- sonable accuracy, even applying to a different boundary condition type.
CONCLUSION
Spatiotemporal dynamics often can be described by means of advection-diffusion type equations, such Burger's, siffusion-sorption, or diffusion-reaction equations. When modeling those dynam- ies with ANNs, large efforts must be taken to prevent the models from overfitting (given the model able to learn the problem all). The incorporation of physical knowledge can be seen as regular- ization, yielding robust predictions beyond the training data distribution.
With FINN, we have introduced a modular, physics- neural network with excellent generaliza- tion abilities beyond different initial and boundary conditions, when contrasted to pure ML models and other physies-aware models. FINN is able to model and extract unknown constituents of differ- ential equations, allowing high interpretability and an assessment of the plausibility of the model's -otto---iitribution predictions. As next steps we seek to apply FINN beyond second order spa- tial derivatives, improve its scalability to large datasets, and make it applicable to heterogeneously distributed data (i.e. represented as graphs) by modifying the module Φu to approximate variable and location-specifi stencils (for more details limitations of FINN, the reader is referred to to|sub- section A.2). Another promising future direction is the application of FINN to worldworld weather data.

---

Under review as a conference paper at ICLR 2022
AUTHOR CONTRIBUTIONS
Hidden for anonymization reasons.
ACKNOWLEDGMENTS
We thank the reviewers for suggesting the evaluation of polynomial fitting and least squares regres- sion as additional baselines as well as modeling another challenging equation (we chose Allen-Cahn with its cubic term) to demonstrate the wide applicability of the method, which we have added the appendix. The results confirm our claims and further suggestions helped us increase the comprehensibility and clarity of the method. Additional parts are hidden for anonymization reasons
REPRODUCIBILITY STATEMENT
Code and data that used for this paper be found in the repository ***** [hidden for anonymization reasons. Instead, the supplementary material of this submission contains an anony- mous version of the repository's README. mil along with the according data and model scripts.
REFERENCES
Y. Bar-Sinai, S. Hoyer, J. Hickey, and M.P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 16631))15334-15549, 2019.
C. Basdevant, M. Deville, P. Haldenwang, J.M. Lacroix, J. Ouazzani, R. Peyret, P. Orlandi, and A.T. Patera. Spectral and finite difference solutions of the burgers equation. Computers & Fluids, 14 (1):23-41, 1986. doi: https://doi.org/ org/ 10. (116/0045-793(86690000
P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez- Ganzazz-Gonzalzz, Zambaldi, M. Malinowski, A. Tac- chetti, D. Raposo, A. Santoro, R. Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint
2018.
J.C. Butcher. Numerical Methods for Ordinary Differential Equations. Wiley, 2008. ISBN 9780470753750.
R.T.Q. Chen, Y. Rubanova, J. Bettencourt, and D.K. Duvenaud. Neural ordinary differential equa- tions. arXiv preprint 1 800607366606... 2018.
R. Courant, K. Friedrichs, and H. Lewy. On the partial difference equations of mathematical physics. IBM Journal of Research and Development, 11(2):115-234, 1967. doi: 10.1147/rd. 112.0215.
G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 204):303-314, 1989.
B. Fornberg. Generation of finite difference formulas on arbitrarily spaced grids. Mathematics of computation, 51(184):699-706, 1988.
V.I.. Guen and N. Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474-11484, 2020.
E. Isaacson and H.B. Keller. Analysis of Numerical Methods, Dover Books on Mathematics. Dover Publications, 1994. ISBN 9780486680293.
N. Kalchbrenner, L. Espeholt, K. Simonyan, A. Oord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610. 10099, 2016.
M. Karlbauer, S. Otte, H.P.A. Lensch, T. Scholten, V. Wulfmeyer, and M.V. Butz. A distributed neural network architecture for robust non-linear spatio-temporal prediction. arXiv preprint 1912 2019.
10

---

Under review as a conference paper at ICLR 2022
G.A. Klaasen and W.C. Troy. Stationary solutions of system of ceaction-diffusiie equations derived from the fitzhugh nagumo equations. SIAM Journal on Applied Mathematics, 44(1): 96-110, 1984. doi: 10.1 137/0144008.
D. Kochkov, J.A. Smith, A. Alieva, Q. Wang. M.P. Brenner, and S. Hoyer. Machine learning- accelerated computational fluid dynamies. Proceedings of the National Academy of Sciences, 118 (21), 2021.
B.M. Lake. Compositional generalization through meta sequennce-o--eeuueee learning. In Advances in Neural Information Processing Systems 32, pp. 9791-9801. 2019.
B.M. Lake, T.D. Ullman, J.B. Tenenbaum, and S.J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2017. doi: 10. 6001888777
Z. Li, Kovachki, K. Arizzadenesheli, B. Liu, Bhattacharya, Stuart, and Anand- kumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020a.
Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandku- mar. Neural operator: Graph kernel network for partial diffcrential equations. arXiv preprint wrXiv:2003.03485, 2020b.
Z. Long, Lu, X. Ma, and B. Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216. PMLR, 2018.
F. Moukalled, L. Mangani, and M. Darwish. The Finite Volume Method in Computational Fluid Dynamics. Springer, I edition, 2016. doi: 10.1007997--3-319-16677-4
W. Nowak and A. Guthke. Entropy-based experimental design for optimal model discrimination in the geosciences. Entropy, 18(11), 2016. doi: 10.3390/e 1814099.. 10409.
T. Praditia, T. Walser, S. Oladyshkin, and W. Nowak. Improving thermochemical energy storage dy- namics forecast with physics-inspired neural network architecture. Energies, 13(15):3873, 2020. doi: 10.3399/en13153887.
Praditia, M. Karlbauer, S. Otte, Oladyshkin, M.V. Butz, and Nowak. Finite volume neural network: Modeling subsurface contaminant transport. arXiv preprint arXiv:2104.06010 2021.
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019. doi: 10. 60j.0.//j..p.... jcp.2018 8. 10.045.
K.F. Riley, M.P. Hobson, and S.J. Bence. Mathematical Methods for Physics and Engineering. Cambridge University Press, 2009. ISBN 9780521 1338888
S. Seo, C. Meng, and Y. Liu. Physics-aware difference graph networks for sparsely-observed dy- namics. In International Conference on Learning Representations, 2019.
X. Shi, Chen, H. Wang, D.Y. Yeung, W.K. Wong, and W.C. Woo. Convolutional 1stm network: A machine learning approach for precipitation nowcasting, arXiv preprint
2015.
Sitzmann, J. Martel, A Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
P.K. Smolarkiewicz. simple positive definite advection scheme with small implicit diffusion. Monthly Weather Review, 111(3):479 - 486, 1983. doi:
A. Turing. The chemical basis of morphogenesis. Philosophical Transactions of the Royal Society B, 237:37-72, 1952.
H.K. Versteeg and W. Malalasekera. An Introduction to Computational Fluid Dynamics: The Finite Volume Method. New York, 1995. ISBN 9780470235157.

---

Under review as a conference paper at ICLR 2022
Y. Yin, V.L. Guen, J. Dona, I. Ayed, E. de Bézenac, N. Thome, and P. Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting, arXiv preprint arXiv:2010.04455, 2020.
J. Zhuang, D. Kochkov, Y. Bar-Sinai, M.P. Brenner, and S. Hoyer. Leamndd discretizations for passive scalar advection in a two-dimensional turbulent flow. Physical Review Fluids, 6(6):064605, 2021.
A APPENDIX
This appendix is structured as follows: The detailed differences with the benchmark models used in this work are presented in |subsection A. Then, limitations of FINN are discussed briefly in subsection A.2] Detailed numerical derivation of the first and second order spatial derivative is shown in /uusecttonn A.3] Additional details on the equations used as well as model specifications and in-depth results presented in lubssecction (Burger' (inrger'ss,,suuseeee, subsecbsen A.5 cnaffussion-oopttionn subsection A.6 (diffusion-reacttoon
andisubsection A.7 AAlen-Cahh). Additional polynomial-fitingg baselines account for the equations are reported in (susectton A.8, including ablation where we replace the neural network modules of FINN by polynomials. A robustness analysis of FINN can be found in subsectiion A.9 where our method is evaluated on noisy data from all equations. Results of training PINN with high-resolution data are reported in (subsection A. whereas results of the original PhyDNet architecture are outlined in sssbsecction A.11] The soil parameters and simulation domain used in the diffusion sofffuiionnsorpionn laboratory experiment are presented in subsectiion A.
A.1 DISTINCTION TO RELATED WORK
Pure ML models Originally, FINN was inspired by DISTANA, a pure ML model proposed by Karlbauer al. (2019). While DISTANA has large similarities to Shi et al. (2013)'s 's ConvLSTM, it propagates lateral information through an additional latent map instead of reading lateral infor- mation from the input map directly, as done in CSnVLSTM-ann transforms that lateral information by means of a user-defined combination of arbitrary layers. Additionally, the processing of the two- point flux approximation in FINN using the lateral information, is more akin DISTANA, albeit motivated and augmented by physical knowledge.
Accordingly, the lateral information flow is guaranteed to behave in a physically plausible manner. That is. quantity can either be locally generated by a source (increased), locally absorbed by a sink (decreased), or spatially distributed (move to neighboring cells). Terminology separates the spatially distribution of quantity into diffusion and udvection. Diffusion describes the equalization of quantity from high to low concentration levels, whereas advection is defined as the bulk motion of a large group of particles/atoms caused by external forces. FINN ensures the conservation of laterally propagating quantity, i.e. what flows from left to right will effectively cause decrease left and an increase right. Pure ML models (without physical constraints) cannot be guaranteed to adhere to these fundamental rules.
PINN Since MI models guided by physical knowledge not only behave empirically plausible but also require fewer parameters and generalize to much broader extent, numerous approaches have been proposed recently to combine artificial neural networks with physical knowledge. |Raisii et al. (2019) introduced the physics-informed neural network (PINN), a concrete and outstanding model that explicitly learns provided PDE. As result, PINN mimics e.g. Burger's equation (see equation 7 by learning the quantity function u(z t) for defined initial and boundary conditions with a feedforward network. The partial derivatives are implicitly realized by automatic differentiation of the feedforward network (representing u) with respect to the input variable x or l. The learned neural network thus satisfies the constraints defined by the partial derivatives.
This method has the advantage that it explicitly provides a solution of the desired function u(x, t) and, correspondingly, predictions can be generated for an arbitrary combination of input values, circumventing the need for simulating the entire domain with e.g. carefully chosen simulation step size. In contrast, FINN does not learn the explicit function u(z, t) defined for particular initial and boundary conditions. but approximates the distinct components of the function. These compo- nents are combined as suggested by the physical equation to result in a compositional model that is more universally applicable. That is, in stark contrast to PINN, the compositional function learned
12

---

Under review as a conference paper at ICLR 2022
by FINN can be applied to varying initial and boundary conditions, since the learned individual components provide the same functionality as the corresponding components of the PDE when pro- cessed by numerical solver. Applying the numerical solver, i.c. the finite volume method (FVM), however, requires either complete knowledge of the equation or careful calibration of the unknowns by choosing equations from a library of possible solutions and tuning the parameters. FINN's data driven component can reveal unknown relations, such as the retardation factor of a function, without the need for subjective prior assumptions.
Learning derivatives via convolution An alternative and much addressed approach of implanting physical plausibilty into artificial neural networks is to implicitly learn the first n derivatives using appropriately designed convolution kernels (Long et al. 2018 Li et al. 20200a b Guen & Thome, 2020 Yin et al. 2020; [Sitzmann et al. (2200. These methods exploit the link that most PDEs, such as u C 2,, y/ ....)) can be reformulated as
ou
Du du Ou Ou
δt =
F t. (t.r,y..... u. Ox Ordy' dx2 dr."....
(6)
When learning partial derivatives up to order and setting irrelevant features to zero, these methods have, in principle, the capacity represent most PDEs. However, the degrees of freedom in these methods are still very high and can fail to safely guide the ML algorithm. FINN accounts for the first and second derivative by learning an according stencil in the module fou and combining this stencil with the case- sensitie ReLU module R, which allows a precise control of the information flow resulting from the first and second derivative. More importantly, convolutional structure only allows implementation of Dirichlet and periodic boundary condition (by means of zero- or mirror-padding), and is not appropriate for implementation of other boundary condition types.
Learning ODE coefficients The group around Brenner and Hoyer (Bar-Sinal al. 2019 Kochkov et al. 2021 Zhuang et al. 2021) follow another line of research about learning the co- efficients of ordinary differential equations (ODEs); note that PDEs can be transformed into set of coupled ODEs by means of polynomial expansion or spectral differentiation as shown in Bar- (Sinai et al. (2019). The physical constraint in these works mostly realized by using the temporal derivative as loss.
While this approach share similarities with our method by incorporating physically motivated in- ductive bias into the model, it uses this bias mainly to improve interpolation from coarse to finer grid resolutions and thus accelerate simulation. Our work focuses on discovering unknown rela- tionships/laws (or re-discovering laws in the case of the synthetic examples), such as the advective velocity in the Burger's example, the retardation factor function the diffusiin--orppionn example, and the reaction function in the diffusion-reaacton example. Additionally, the works by Brenner and Hoyer employ convolutional structure, which is only applicable to Dirichlet or periodic boundary conditions, and it suffers from slight instability during training when the training data trajectory is unrolled for a longer period. In contrast, FINN employs the flux kernel, calculated at all control volume surfaces, which enables the implementation and discovery of various boundary conditions. Furthermore and in contrast to Brenner and Hoyer, FINN employs the Neural ODE method as the time integrator reduce numerical instability during training with long time series. However and in accord with this line of research, FINN is also able to generalize well when trained with a relatively sparse dataset (coarse resolution), reducing the computational burden.
Numerical ODE solvers Traditional numerical solvers for ordinary differential equations (ODEs) can be seen as the pure-physics contrast FINN. However, these do not have learning capacity to reveal unknown dependencies or functions from data. Also, as shown in (Yin et al.) [0200. a pure Neural ODE without physical inductive bias does not reach the same level accuracy as physics- aware neural network models. This is underlined by our field experiment where FINN reached lower errors on real-world dataset compared to a conventional FVM model.
A.2 LIMITATIONS OF FINN
While the largest limitation of our current method can be seen in the capacity to only represent first and second order spatial derivatives, this is an issue that we will address in follow up work. Still, FINN can already be applied to a very wide range of problems as most equations in fact only
13

---

Under review as a conference paper at ICLR 2022
depend up the second spatial derivative. So far, FINN is only applicable to homogeneously distributed data-we we intend to extend it to heterogeneous data from graphs. Although we have successfully applied FINN to 2D diffusion reaction data, the training time is considerable. While, according to our observations, this appears to be a common issue for physics-aware neural networks, the implementation of custom-convon convolutonn layers could widen this bottleneck with today's 's hardware- accelerated computation of convolution operations.
A.3 LEARNING THE NUMERICAL STENCH.
Semantically, the yP module learns the numerical stencil, that the geometrical arrangement of a group of neighboring cells to approximate the derivative numerically, effectively learning the first spatial derivative du from both (i:-1, Ui] and [u4, U441], which are the inputs to the y.N. and yo.++ Br module, respectively.
The lateral information flowing from u:-1 and u,+1 toward u; controlled by the Р (advective flux, i.e. bulk motion of many particles/atoms that can either move to the left or to the right) and the yPD (diffusive flux, i.e. drive of particles/atoms to equílibrium from regions of high to low concentration) modules. Since the advective flux can only move either to the left or to the right, it will be considered only in the left flux kernel (f(-) ) or in the right flux kernel (fi+), and not both at the same time. The case-sensitive ReLU module R (Eq. 3) decides on this, by setting the advective flux in the irrelevant flux kernel to zero (effectively depending on the sign of the output of 4aa)- Thus, the advective flux is only considered from either U4-1 u,+1 to Us which amounts to the first order spatial derivative.
The diffusive flux, on the other hand, can propagate from both sides towards the control volume of interest and, hence, the second order spatial derivative, accounting for the difference be- tween 14-1 and U441, has to be applied. In our method, this is realized through the combina- tion of the &P and pD modules, calculating the diffusive fluxes d_ ( 2; ().) and 64 (u,, U4+1 (Pp)) inside of the respective left and right flux kernel. The combination of these two deltas in the state kernel ensures the consideration of the diffusive fluxes from left (includ- ing u;-1) and from right (including U;41), resulting in the ability account for the second order spatial derivative.
Technically, the first, i.e. [-1, 1]. and second, i.e. [1, 22 1]. order spatial differentiation schemes are common definitions and derivation can be found, for example, in Fornberg (1988). However, a quick derivation of the Laplace scheme 2,, 1] can be formulated as follows. Define the second order spatial derivative as
pu ~ dx²
(ōn/Or)): (au/Ox)lit
Ar
with (Qu/Ox)) (u,-1 / Ae and (Qu/ (0u/Ox)kkt (14 - 11441)/////00/ Aa.. Then
ju (u:-1 u;)
(u: - us+1) U4-1 20g + us+1
dr²
A=?
Ar
hence the [+1, +1] as coefficients.
A.4 BURGER
The Burger's equation is commonly employed in various research areas, including fluid mechanies.
Data The ID generalized Burger's equation is written as
Ou
Fu 82m D
=(u) dr Oz2'
at
where the main unknown is u, the advective velocity is denoted as v(u) which is function of and the diffusion coefficient is D = 0.01/a. In the current work, the advective velocity function is chosen to be identity function u(u) to reproduce the experiment conducted in the PINN paper (Raissi et al. 2019).
The simulation domain for the train data is defined with 1-11, 1], [0. 1] and is discretized
with N₂ = 49 spatial locations, and N, = 201 simulation steps. The initial condition is defined as u(x, 0)- sin(TT.)). and the boundary condition is defined w(-1, 6) = u(1,() 0.
14

---

Under review as a conference paper at ICLR 2022
TCN
ConVLSTM
DISTANA
PINN
PhyDNet
FINN
0.7
0.7
0.7
0.7
0.7
0.7
-0.7
-0.7
-0.7
-0.7
-0.7
-0.7
X
X
X
Data
Prediction
Figure 10: Prediction mean over ten different trained models (with 95% confidence interval) of the Burger's equation / for the in-dis-test dataset.
In-dis-test data is simulated with æ = [-1, 1] and a time span of t = [1, and N, 401. Initial condition is taken from the train data at t = and boundary conditions are also similar the train data.
The simulation domain for the out-dis-test data is identical with the train data, except for the initial condition that is defined as u(x 0) = sin(..).
Model architectures Both TCN and ConvLSTM designed to have input neuron, hidden layer of size 32, and one output neuron. The lateral and dynamic input- and output sizes of the DISTANA model are set one and hidden layer of size 32 is used. The pure ML models were trained on the first 150 time steps and validated on the remaining 50 time steps of the train data (applying early stopping). Also, to prevent the pure ML models from diverging too much in closed loop, the boundary data are fed into the models as done during teacher forcing. PINN was defined feedforward network with the size of [2, 20, 20, 20, 20, 20, 20, 20, 20, 1] hidden layers, each contains 20 hidden neurons), as reported in the original work by Raissi et al. 2019). PhyDNet was defined with the PhyCell containing 32 input dimensions, hidden dimensions, hidden layer, and the ConvLSTM containing 32 input dimensions, 32 hidden dimensions, hidden layer. For FINN, the modules РP SD, and РA were used, with РLA defined
feedforward network with the size of [1, 10, 20, 10, 1] that takes u as an input and outputs the advective velocity v(u), and yPD as a learnable scalar that learns the diffusion coefficient D. All models are trained until convergence using the L-BFGS optimizer, except for PhetNet, which is trained with the Adam optimizer and learning rate of 10 due to stability issues when training with the L-BFGS optimizer.
Additional results Individual errors of the ten different training runs as well as visualizations for the train (Table 2). in-dis-test (Table and Figure 10), and out-dis-test (Table and Figure IT) datasets.
Table Closed loop MSE on the train data from ten different training runs for each model for the Burger's equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
4.1 10-2
7.9 x 10-2
1.2 10-1
6.1 10-1
8.0 10-5
1.9 10 10-6
2
5.1 x 10
1.7 10-1
1.8 X 10
4.1 X 10
4.7 x 10
1.5 X 10
3
3.6 x 10
1.3 10-2
6.4 10-5
4.9 X 10
6.8 10-5
2.2 X 10
4
4.3 x 10-2
2.7 10-4
2.9 10-4
5.6 10-4
1.2 10-4
2.5 x 10
5
5.4 10
2.1 14
2.2x 10
10
6.7 10
1.6 10
6
3.9 x 10
10-3
4
5.2x 10
5.5 1
9.9 x 10
7
6.6 10
10
10
5.2 1
5.1 10
2.2 x 10 10
8
1.2 x 10
10-2
2
4.8 10-4
5.6 10-5
3.0 x 10
9
3.4 x 10
5.5 x 10
3.9 1
5.5 10
9.9 X 10
1.1 10
10
6.8 10
3.2 10
1.1 1
5.0 X -
8.1 1
4.5 10 10-7
15

---

Under review as a conference paper at ICLR 2022
Table Closed loop MSE on in-dis-test data from ten different training runs for each model for the Burger's equation.
Run
TCN
ConVLSTM
DISTANA
PINN
PhyDNet
FINN
3.0 10
1.2 x 10
2.6 10
4.1 X 10-3
1.1 10
1.6 10
2
7.0 x 10-2
3.7 x 10-1
1.1 10-3
1.3 10-3
2.5 10-1
1.3 X 10
3
2.4 x 10-2
2.0 x 10-1
8.9 10-1
x 10-3
3.4 x 10-2
1.9 x 10
5.1 10-2
5.6 10-3
6.2 10-3
30-3
4.7 10
2.1 10
5
5.4 x 10-2
2.6 x 10-3
x X 10-3
2.6
2 10-3
1.1 10
1.3 X 10
6
3.2 x 10
1.0 X 10-2
1.2 10
1
3.2 10
6.8 x 10
8.8 10-2
5.5 x 10-3
1.1 10-3
10-3
3.3 10
1.8 10-6
8
1.1 x 10"
1.6 x 10
4.0 X 10-3
x 10
2.7 10
2.4 10
9
3.3 x 10-2
1.9 x 10-1
4.0 X 10-3
3.0 X 10-2
2.6 10
1.2 X 10
10
3.6 10
9.8 10
6.0 X 10-3
1,5 x 10-3
8.0 10
2.8 10
Table 4: Closed loop MSE on out-dis-test data from ten different training runs for each model for the Burger's equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
3.6 X 10
7.2 x 10-2
5.9 X 10-3
5.3 x 10-2
1.9 X 10
2
3.4 x 10
1.9 x 10-1
4.4 X 10
2.1 10
1,5 X 10
3
3.3 x 10-2
2.9 x 10-3
2.9 X 10-3
4.5 x 10
2.2 X 10
4
3.7 x 10-2
8.9 x 10-4
14 10-3
2.0 x 10
2.5 x 10-6
5
4.2 10
1.2 10
1.0 10
3.1 X 10
1.6 10 10-6
6
2.7 x 10
1.4 10
3.0 X 10
1.1 X 10
9.9 X 10
7
6.7 x 10
3.3 x 10-2
6,2 X 10
2,8 x 10
2.2 x 10
8
1.1 10
4.0 10-2
7.0 X 10-4
4.7 10
3.0 10 10-6
9
3.0 x 10
8.5 x 10
7.4 X 10
4.5 10
1.1 X 10
10
1.2 10
3.1 x 10
7.0 X 10
4.5 10
4.5 X 10 10
TCN
ConvLSTM
DISTANA
PhyDNet
FINN
0.7
0.7
0.7
0.7
0.7
0
0
-0.7
0077
-0.7
-0.7
-0.7
-1
0
1
1
0
1
0
-1
-1
X
X
X
X
Data
Prediction
Figure 11: Prediction mean over ten different trained models (with 95% confidence interval) of the Burger's is equation at t = for the out- out-dis-test data.
16

---

Under review as a conference paper at ICLR 2022
A.5 DIFFUSION-SORPTION
The diffusion-sorption equation is another widely applied equation in fluid mechanics. A practical example of the equation to model contaminant transport in groundwater. Its retardation factor R can be modelled using different closed parametric relations known as sorption isotherms that should be calibrated to observation data.
Data
The ID diffusion-sorpitn equation is written as the following coupled system of equations:
0u
Su R(u) Ox.²*
D
du ō1
8²u DΦ
8
0t
dx2
where the dissolved concentration and total concentration are the main unknowns. The effective diffusion coefficient denoted by D 10-4, the retardation factor is R(uu,, function u, and the porosity is denoted by Φ = 0.29.
In this work, the Freundlich sorption isotherm, see details in Nowak & Guthke 2016). was chosen to define the retardation factor:
R(u) = + 1 Φ Paknyuu
(10)
where Ps = 2 880 is the bulk density, 3.5 10 the Freundlich's parameter, and n = 0.874
the Freundlich's 's exponent.
The simulation domain for the train data defined with æ = =0, 1).6= [0, 500] and discretized with N₂ 26 spatial locations, and N, 501 simulation steps. The initial condition is defined as uz 0) = 0, and the boundary condition is defined as u(0, t) 1.0 and u(1, t) axe
The in-dis-test data simulated with [0, and with the time span of [2 500, 10 000) and N₂ = 2001. Initial condition is taken from the train data at 500 and boundary conditions are also identical the train data.
The simulation domain for the out-dis-fest data was identical with the in-dis-test data, except for the boundary condition that was defined as u(0, = 0.7.
Model architectures TCN is designed to have two input neurons, four hidden layers of size [4, 8, 8), and two output neurons. ConvLSTM has two input- and output neurons and hidden layer with 16 neurons. The lateral and dynamic input- and output sizes of the DISTANA model set to one and two, respectively, while a hidden layer of size 12 used. The pure ML models were trained on the first 400 time steps and validated on the remaining 100 time steps of the train data (applying early stopping). Also, prevent the pure ML models from diverging too much in closed loop, the boundary data are fed into the models as done during teacher forcing. PINN was
Table Closed loop MSE on the train data from ten different training runs for each model for the diffusion-sorptiion equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
3.6 10-2
4.8 x 10-5
2.1 10-5
2.2 10-1
3.1 10-5
1.1 10-4
2
2.0 x 10
2.3 x 10-2
1.6 X 10-5
2.3 X 10
2.5 10
1.4 X 10
3
8.1 x 10-2
4.4 10-2
2.7 x 10-5
x.2 10-6
3.2 10
1.1 X 10
4
2.0 10-1
1.0 10-2
9.3 10-5
2.1x 10-1
2.9 10-5
4.1 X 10
5
1.8 x 10
8.7 x 10
6,2 10
2
7.7 10
5.3 X 10
6
2.7 x 10
5.9 x 1
x.0
x
4.4 10
1.9 X 10
7
3.2 10-4
6.4
1
8.6 10
1.8 10
1.5 x 10
8
1.7 x 10
1.4 10
2.5 10-5
2.0 X 10
2.4 10-5
1.2 10
9
4.3 x 10
2.2 x 10
5.9 10
9.0 10
2.1 10
1.2 X 10
10
6.6 10
2.7 10
3.9 1
1.1 X 1
5.2 1
1.7 10 10-5
17

---

Under review as a conference paper at ICLR 2022
Table Closed loop MSE on in-dis-test data from ten different training runs for each model for the ceffusiion-sopttion equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
1.9 x 10-2
7.7 x 10-3
6.9 X 10
3.0 10-2
3.8 10 10-3
3.2 10
2
2.4 x 10-1
1.0 x 10-2
8.5 x 10-5
8.6 10
8.4 10
3.5 X 10
3
4.7 10
2.1 10
1.7 10-3
9.3 10-5
2.9 10
3.2 x 10
4
2.4 x 10-1
2.8 10-2
x 10-3
5.2x100
3.6 10
1.1 10
5
3.5 x 10-2
7.9 X 10-2
7.3 x 10-3
7 10-6
2.8 10
9.8 x 10
6
2.4 x 10
x.4 10-2
3.1 10
1 10-5
4.8 10
4.9 10
7
1.4 10-3
4.2 10-2
6.5 10-3
1.8 10
1.0 10-3
3.7 10
8
3.9 x 10
2.2 x 10
9.4 X 10
6.9 X 10
2.5 10
3,4 10
9
5.6 x 10
1 10
x x 10
4.5 x 10
2.4 10
2.8 10
10
3.4 x 10
4.2 10
9.9 10
3.5 10
2.8 10
4.1 10
TCN
ConVLSTM
DISTANA
PINN
PhyDNet
FINN
0.5
0.5
0.5
0.5
0
0-
0.5
0.5
0.5
0.5
0.5
0.5
x
Dala
Prediction
Figure 12: Prediction mean over ten different trained models (with 95% confidence interval) of the dissolved concentration in the ceffiusion-sorptiin equation at 10000 for the in-dis-test dataset.
Table 7: Closed loop MSE on out-dis-test data from ten different training runs for each model for the ceffusiion-soppticn equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
4.7 10
2.3 10
2.9 10-3
1,3 10
9.7 10
2
2.1 x 10-1
4.3 x 10-2
1.6 x 10-3
1.1 10-2
1.5 x 10
3
9.2 x 10-2
5.5 x 10-2
3.4 10-3
1.4 10-2
9.5 x 10-5
2.0 10-1
1.9 10
2.0 10
1.1 10
3.4 10-5
5
4.4x 10-2
1.3 x 10-1
4.6 10-4
2.1 10
4.9 X 10
6
3.9 x 10-4
1.0 x 10-1
8.7 X 10-3
4.4 10-2
1.9 X 10
9.3 10-4
9.1 x 10-2
1.6 10-2
1.5 10-2
1.5 10-5
8
5.0 10
2.6 x 10-3
9.5 X 10
1.2 10
1.0 X 10
9
4.6 x 10-1
3.4 10-2
7.1 X 10-3
1.3 x 10
1.2 x 10
10
4.1 10
8.0 x 10
2.6 X 10
1.3 10
1.6 10
TCN
ConvLSTM
DISTANA
PhyDNet
FINN
1
1
0.5
0.5
0
0
0
0
0.5
0
0.5
1
0
0.5
0.5
0.5 X
X
X
x
Data
Prediction
Figure 13: Prediction mean over ten different trained models (with 95% confidence interval) of the dissolved concentration in the diffusion-sorption equation at 10 for the out-dis-tes dataset.
18

---

Under review as a conference paper at ICLR 2022
defined feedforward network with the size of [2. 20, 20, 20, 20, 20, 20, 20, 20, 2]. PhyDNet was defined with the PhyCell containing 32 imput dimensions, hidden dimensions, 1 hidden layer, and the ConvLSTM containing 32 input dimensions, 32 hidden dimensions, 1 hidden layer. For FINN, the modules yP and pp were used, with ypD defined as a feedforward network with the size input and ouiputs the retardation factor R(u). All models trained until convergence using the L-BFGS optimizer, except for PhyDNet, which is trained with the Adam optimizer and learning rate of x 10 due stability issues when training with the L-BFGS optimizer.
of [1, 10, 20, 10, 1] that takes
Additional results Individual errors of the ten different training runs as well as visualizations for the train (Table 5}. in-dis-test (Table 6 and Figure 12), and out-dis-test (Table 7 and FFigure 13) datasets. Results for the total concentration d, were omitted due to high similarity to the concentra- tion of the reported contamination solution
A.6 CNFFUSION-REACTOON
The diffussio-reeaction equation is applicable in physical and biological systems, for example in pattern formation Turing 1952).
Data the current paper, we consider the 2D diffusion--eaction for that class of problems:
Ou = Ri (wi, (2) + Dj 0t
(
(8u, Ox?
Ouu 0y
Ou2
88u2
82u2
R2(u1, u2) + D2
+
Or.
dy²
(11)
(12)
Here, D1 = 10 and D2 x 10-3 are the diffusion coefficient for the activator and in- hibitor, respectively. The system equations is coupled through the reaction terms R1 (1(1 uz) and R2 (11 U2] which are both dependent on मN and In this work, the Fizzhugh-Ngumo (Klaasen Troy 1984) system was considered define the reaction function:
(u, u2) = un
(13)
R2(u1, U2) = U22
(14)
with & x 10-3
The simulation domain for the train data is defined with x (-1, 1], 1- [-1, t= [0, 10] and discretized with N₂ = 49 and N, 49 spatial locations, and N, = 101 simulation steps. The
initial condition was defined as (v. 0) u2 (2r, 0) = sin(T((x + 1) sin((i( 11/2), and the dus
corresponding boundary condition was defined as dz 1, 0. dw and Oug (1, t) O.
0u (1, 0. Oug 1, 0.
Table 8: Closed loop MSE on train data from ten different training runs for each model for the diffusion-reactiinn equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
8.7 x 10-3
1.7 x 10-3
1.7 x 10-3
2.7 10-1
8.5 10-5
1.0 10-4
2
4.4 x 10
1.6 x 10-3
4.2 X 10-3
5.6 X 10
7.1 x 10
1.6 x 10
3
3.2 x 10
1.4 x 10-3
9.1 10-3
2.1 10-4
5.8 x 10-5
1.4 X 10
4
8.7 x 10-3
7.8 10-4
6.5 10-4
2.8
6.8 10-5
1.2 10
5
2.5 x 10
1,5 10
x.7x
x.8
7.6 10
1.0 X 10
6
8.0 x 10
108110
10
x0
6.5 1
1.7 X 10
7
2 1
x0
16x10-1
60-110-5
8.2 10
1.6 10
8
1.1 x 10
10
24x10-8
1
7.8 x 10-5
1.0 X 10
9
1.3 x 10
2.2 x 10
3.8 x 10
3.2 10
8.8 10
1,5 X 10
10
5.6 10-3
7.3 10
8.6 1
6.2 10
7.4 1
1.1 X 10
19

---

Under review as a conference paper at ICLR 2022
Table Closed loop MSE on in-dis-test data from ten different training runs for each model for the diffusion-reaction equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
1.8 10
8.0 x 10-2
1.6 X 10-1
7.7 10
6.2 X 10
2.9 10
2
5.9 x 10-1
5.7 x 10-2
1.9 X 10-1
2.3 10
8.8 10
1.6 10
3
6.5 10
9.2 10-2
3.0 10
3 10-2
7.8 10
1.7 10
4
3.1 x 10-1
6.7 10-2
2.0 10-1
20-2
6.8 60-2
2.9 X 10
5
7.3 x 10
4.9 X 10-2
1.9 10-1
1 10-1 10
5.3 10
1.7 10
6
1.9 x 10
5.4 10
2 10-2
8.9 x 10-2
9.9 10
2.0 10
7
6.0 10-1
1.0 10-1
2.2 10-1
x 10-2
6.7 10-2
1.5 x 10
8
7.1 x 10
8.6 x 10
1.9 10
1,3 x 10
9.1 10
2.3 X 10
9
5.5 x 10
2.3 10
1.6 10
2.3 10
6.7 10
1.8 10
10
2.2
10
1.2 10
1.7 X 10
2.6 10
1.1 10
2.1 10
TCN
ConVLSTM
DISTANA
PINN
PhyDNet
FINNS
mmr
X
Dala
Prediction
Figure 14: Prediction mean over ten different trained models (with 95% confidence interval) of the activator the diffusion-reaction equation 50 for the in-dis-test dataset.
Table 10: Closed loop MSE on out-dis-test data from ten different training runs for each model for the diffusion-reactioo equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
2.3 10 2
3.2 10
9.4 10-3
3.9 10
6.5 10
2
1.7 x 10-1
1.3 10-2
1.3 x 10-2
5.4 x 10-2
5.5 X 10
3
2.5 x 10-1
1.3 10-2
2.1 10-2
2.5 x 10-2
5.9 x 10
7.3 10-2
7.2 10-3
7.9 10-3
4.2 10
6.5 10-3
5
2.6 x 10-1
5.9 10-3
2 X 10-3
2.8 10
6.1 X 10
6
6.4 x 10-2
1.5 x 10-2
2.1 10-3
2.9 10-2
6.0 X 10
2.2 x 10-1
1.0 10-2
1.9 10-2
2.8 10-2
5.7 10-3
8
2.0 10
3.9 X 10-3
1.2 10
2.1 10
6.1 10
9
2.3 x 10
4.3 10-2
1.3 X 10-2
2.1 x 10
6.0 X 10
10
3.6 10
3.7 10
3.3 X 10
5.9 10
6.2 10
TCN
ConvLSTM
DISTANA
PhyDNet
FINN
0.6
0.6
0.6
0.6
0.6
0.3
0.3
0.3
0.3
0.3
-0.3
-0.3
00.3
0.3
00.3
-0.6
-0.6
-0.6
0.66
-0.6
1
0
1
0
1
1
0
X
X
X
Data
Prediction
Figure 15: Prediction mean over ten different trained models (with 95% confidence interval) of the activator in the diffusion-reaction equation at t 10 for the out-dis-test dataset.
20

---

Under review as a conference paper at ICLR 2022
The in-dis-test data is simulated with with [-,,1], y [-1,1] and with the time span of t = [10, 50] and N, = 501. Initial condition taken from the train data at t = 10 and boundary conditions also identical the train data.
The simulation domain for the out-dis-test data was identical with the train data, except for the initial condition that was defined u1 (a:, 0) "2(2; 0) sin(r( + 1)/2) sin(x(y 1)/2) 0.5, Le. subtracting 0.5 from the original initial condition.
Model architectures TCN designed have two input- and output neurons, and one hidden layer of size 32. ConvLSTM has two input- and output neurons and one hidden layer of size 24. The lateral and dynamic input- and output sizes of the DISTANA model are set to one and two, respectively, while a hidden layer of size 32 is used. The pure ML models were trained on the first 70 time steps and validated on the remaining 30 time steps of the train data (applying early stopping). Also, to prevent the pure ML models from diverging too much in closed loop, the boundary data are fed into the models as done during teacher foreing. PINN is defined as a feedforward network with the size of [3, 20, 20, 20, 20, 20, 20, 20, 20, 2]. PhyDNet is defined with the PhyCell containing 32 imput dimensions, 49 hidden dimensions, hidden layer. and the ConvLSTM containing 32 input dimensions, 32 hidden dimensions, hidden layer. For FINN, the modules yP,, yD and & used, with (pD set as two learnable scalars that learn the diffusion coefficients D1 and D2, and & defined as feedforward network with the size of [2, 20, 20, 20, that takes u and ug as inputs and outputs the reaction fuunctions Rí (u1, U2) and R2 (w) u2). All models are trained until convergence using the L-BFGS optimizer, except for PhyDNet, which is trained with the Adam optimizer and learning rate of 1 10 due to stability issues when training with the L-BFGS optimizer.
Additional results Individual errors of the ten different training runs as well as visualizations for the train TTble 8), in-dis-test (Talle 9 and Figure 14), and out-dis-test (Table 10 and Figure 15) datasets. Results for the total inhibitor U2 were omitted due to high similarity to the reported activator W1.
A.7 ALLEN-CAHN
The Allen-Cahn equation is commonly employed in reaction-diffusion systems, in particular to model phase separation in multi-component alloy systems (Raissi et al. 2019)
Data
The ID Allen-Cahn equation is written
du 8 Or + R(u),
D
(15)
at
where the main unknown is u, the reaction term is denoted as R(u) which is a function of a and the diffusion coefficient is 10- In the current work, the reaction term defined as:
R(u) 5u 5u.
(16)
Data
TCN
ConVLSTM
DISTANA
PINN
PhyDNet
FINN
0-
0
0.5
-1
-1
-1
-1
-1
-1
-1
0.5
t
1
0-
0
0
0
0
11
0
0
Data
Prediction
Figure 16: Plots of Allen-Cahn's data and prediction of in-dis-test data using different models. The plots in the first row show the solution over æ and t (the red lines mark the transition from train to in-dis-test), the second row visualizes the best model's solution distributed in at t 1.
21

---

Under review as a conference paper at ICLR 2022
Data
TCN
ConVLSTM
DISTANA
PhyDNet
FINN
0.5
0
0
0
0.5
0
0.5
1
0
0.5
1
0.5
1
0.5
0.5
t=1
t=
1
t
1
1
0
-1
1
-1
-1
-1
0
Data
Prediction
Figure 17: Plots of Allen-Cahn's data and prediction of out-dis-test data using different models.
The plots in the first row show the solution over x and the second row visualizes the solution distributed in x at t = 1. Right wagga
to reproduce the experiment conducted in the PINN paper (Raissi et al. 2019).
The simulation domain for the train data defined with x = [-1, 1]. = [0, 0.5) and discretized with N₂ 49 spatial locations, and N, 201 simulation steps. The initial condition is defined as u(2 0) = x²² cos(TT)) and periodic boundary condition is used, i.c. u( ((-1,t) = u(1,t).
In-dis-test data is simulated with [-1, and time span of [0.5, 1] and N, 401. Initial condition is taken from the train data at /. = 0.5 and boundary conditions are also similar to the train data.
The simulation domain for the out-dis-test data is identical with the train data, except for the initial condition that is defined u(x sin(Tx/22) /2).
Model architectures Both TCN and ConvLSTM are designed to have one input neuron, one hidden layer of size 32, and one output neuron. The lateral and dynamic input- and output sizes of the DISTANA model are set to one and hidden layer size 32 is used. The pure ML models trained the first 1.50 time steps and validated the remaining 50 time steps of the train data (applying early stopping). Also, to prevent the pure MI models from diverging too much in closed loop, the boundary data fed into the models as done during teacher foreing. PINN was defined as feedforward network with the size of [2, 20, 20, 20, 20, 20, 20, 20, 20, 1] (8 hidden layers, each contains hidden neurons). PhyDNet was defined with the PhyCell containing 32 input dimensions, 7 hidden dimensions. 1 hidden layer, and the ConvLSTM containing 32 input dimensions, 32 hidden dimensions, hidden layer. For FINN, the modules PA., PD, and 4 were used, with yp defined as learnable scalar that learns the diffusion coefficient D, and oy defined
Table Closed loop MSE on the train data from ten different training runs for each model for the Allen-Cahn equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
1.1 10-2
7.1 x 10-2
2.9 10-1
1.6 10-5
1.5 10-1
1.8 x 10-5
2
4.5 x 10
3.0 x 10-2
4.7 X 10
4.2 10
6.4 10
8.2 x 10
3
2.9 x 10
1.2 x 10-2
8.0 X 10-4
x 10-5
2.5 10-4
5.4 10
4
8.9 10-2
6.5 x 10-3
4.9 10-4
1.6 10-6
4.8 10-5
4.7 10-7
5
1.4 x 10 1
8.0 x 10
2.6 10
1.3110
5.1 10
2.3 x 10
6
1.1 x 10
4.6 10
10
6 10
4.7 10
2.5 x 10
7
9.6 x 10-3
7.1 10
10-10-4
1.4 10-5
1.7 10-4
1.1 10
8
3.6 x 10
5.3 x 10-4
8.5 10-4
211100
3.7 10-5
2.9 X 10
9
6.9 x 10
4.0 x 10
7.6 1
2.0 10-5
9.8 10
3.0 X 10
10
2.8 10-2
1.8 10
1.5 1
x -
9.8 1
2.9 10 10-7
22

---

Under review as a conference paper at ICLR 2022
Table 12: Closed loop MSE on in-dis-test data from ten different training runs for each model for the Allen-Caht equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
9.9 x 10
1.2 X 10
4.8 X 10
7.0 X 10 3
1.4 X 10
5.6 X 10 5
2
2.9 10
8.6 x 10
9.1 10
1.8 10
8.2 10
2,4 10
3
2.3 x 10-1
3.6 x 10-1
7.9 10-2
x 10
1.0 10
1.6 x 10
4
4.7 x 10
x.4 10-1
7.4 10-2
x.6x1000
9.9 X 10
2.1 10-6
5
1.9 x 10
5.5 10
2
8.5 10
9.3 10-5
6
1,2 10
3.2 10
1.0 10-1
2.0 20-3
1,2 12XI 10
1.1 10
7
1.2 x 10
5.4 10-1
1.3 1.3 10-1
5.0 10-4
2.3 X 10
x.4 10
8
3.0 10-1
5.6 10-1
9.5 10-2
1.3 10-1
10
1.3 10
9
1.4% x 10
1.3 x 10
1.6 X 10-1
x 10
1.0 10
1.3 x 10
10
2.3 x 10
X.5 X 10
4.6 X 10
3.3 X 10
9.4 X 10
1.3 X 10
ConvLSTM
PhyDNet
2
2
1
-1
11
1
0
11
0
X
Data
Prediction
Figure 18: Prediction mean over ten different trained models (with 95% confidence interval) of Allen-Cahn equation at / for the in-dis-test dataset.
Table 13: Closed loop MSE on out-dis-test data from different training runs for each model for the Allen-Cahn equation.
Run
TCN
ConvLSTM
DISTANA
PINN
PhyDNet
FINN
1
4.1 x 10
5.3 x 10
1.4 X 10
6.2 10
7.4 X 10
2
2.0 x 10-1
3.8 x 10-1
3.7 X 10-2
2.4 x 10-1
2.9 X 10-5
3
1.3 10-1
1.5 x 10-1
6.2 10
6.2 10
1.8 x 10
4
2.0 x 10-1
2.9 x 10-1
1.1 10
7.3 10
3.0 X 10
5
1.7 x 10-1
8.3 10
X x 10
6.8 x 10
1.2 x 10
6
8.7 10
2.9 10
3.1x
7.8 10
1.5 10-5
7
1.0 x 10
1.5 x 10
5.9 X 10
1.0 x 10
4.3 x 10
8
1.5 x 10
5.3 x 10-1
9.1 X 10
5.4 x 10
1.9 X 10
9
1.2 10
1.4 x 10
1.0 10-1
10-1
1.8 10
10
9.6 10-2
x x 10
x X 10
5.9 10-1
1.9 10
COnVLSTM
PhyDNet
-1
-1
Prediction
Dota
Proditioon
Figure 19: Prediction mean over ten different trained models (with 95% confidence interval) of Allen-Cahn equation at / 1 for the out-dis-test data. The right-most plot shows the accurately learned reaction function from the data samples.
23

---

Under review as a conference paper at ICLR 2022
feedforward network with the size of [1, 10, 20, 10, 1] that takes input and outputs the reaction function R()). All models are trained until convergence using the L-BFGS optimizer, except for PhyDNet, which is trained with the Adam optimizer and learning rate of x 10 due to stability issues when training with the I-BFGS optimizer.
Additional results Individual errors of the ten different training runs are reported as well as visu- alizations are generated for the train (Table I1) in-dis-test (Table 12 Figure 16 and Figure 18), and out-dis-test (Table 13 Figure 17 Figure 19) datasets. Morcover, the righl-most plot in Figure 19 shows the dataset's
A.8 BASELINE ASSESSMENT WITH POLYNOMIAL REGRESSION
In this section, we apply polynomial regression to show that the example problems chosen in this work (i.e. Burger's, diffusion-sorption diffusion-reaction, and Allen-Cahn) are easy solve. First, we use polynomial regression to fit the unknown variable u fax f), similar to PINN. Fig- ure 20 shows the prediction of for each example, obtained using the fitted polynomial coefficients. For the Burger's s equation, the train and in-dis-tess predictions have MSE values of 5.0 x 10 and 4.1 10-2, respectively. For the diffusion-sorptie sorppton equation, the train and in-dis-test predictions have MSE values of 3.0 x 10-3 and 4.1 x 10 respectively. For the diffusion-reaction equation, the train and in-dis-test predictions have MSE values of 1.7 x 10-2 and 3.9 x 10a, respectively.
Burgers', polynomial (d=5)
Diffusion sorption, polynomial (d=3)
Training
Extrapolation
Training
Extrapolation
1.0
1.0
0.5
0.5
0.5
0.5
0.5
-0.5
0.5
-1
-1
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0
2500
0
10000
t
t= 2
2500
10000
0.5
1
0.5
M
20
0.5
10
-0.5
0.5
D
-1
0.5
0
0.5
x
Data
Prediction
Data
Prediction
Diffusion reaction, polynomial (d=5)
Allen-Cahn, polynomial (d=10)
Training
Extrapolation
Training
Extrapolation
1
0.5
0.5
0.0
0.0
-055
-0.5
-0.5
-1
-1
-1
-1
1
0
0.00
0.25
0.50
0.0
0.5
1.0
== 10.
0
50, y
0
t 0.5
t= 1
0.5
700
0.5
300
350
0.5
150
-1.0
0.5
Data
Prediction
Data
Prediction
Figure 20: Prediction pairs for train and in-dis-test (left and right figure columns of the pairs) the Burger's 's (left top, polynomial order 5), diffusion-sorption (top right, polynomial order 3), diffusion- reaction (bottom left, polynomial order 5 since higher orders did not converge), and Allen-Cahi (bottom right, polynomial order 10) equations using polynomial regression.
24

---

Under review as a conference paper at ICLR 2022
Burgers'
Diffusion reaction
Allen-Cahn
0.5
0.0
0.0
00.
0.5
-1
-1
-1
0.0
0.5
1.0
0.00
0.25
0.50
t= 1
t= 10, 0
0.5
0.5
0.5
0.0
5
-0.5
-0.5
-0.5
Prediction
Data
Prediction
Data
Prediction
(a)
(b)
(c)
Figure 21: Plots the train prediction the Burger's s (left), Cahn equations (right) using FINN with polynomial fitting. Due instability issues, the diffusion- sorption equation could not be solved with the polynomial FINN. The plots in the first row show the solution x and I, and the plots in the second show the solution distributed in x.
(center), and Allen-
For the Allen-Cahn equation, the train and in-dis-test predictions have MSE values of 8.8 10-3 and 2.2 x 10S, respectively. The simple polynomial fitting fails to obtain accurate predictions of the solution for all example problems. The results also show that the polynomials overfit the data, evi- denced by the significant deterioration of performance during extrapolation (prediction of in rist data). The diffusion-reactio because they require higher order polynomials to obtain reasonable accuracy. With the high order, they still fail to even fit the train data well.
and the Allen-Cahn equations are particularly the most diffieult to fit,
Next, we also consider using polynomial fitting in lieu of ANNs (namely the modules 9.A. PDD and 2 in FINN. With this method, the model successfully obtain accurate prediction of the Burger's equation Figure 21ap. The MSE values are 1.9 x 10-4, 1.0 10-3 (0-3,_ and 1.3 10 4 for train, in- dis-test, and out-dis-lest data, respectively. However, the model fails to complete the training for the diffusion-sorptie sorption equation due to major instabilities (the polynomials can produce negative output and therefore, negative diffusion coefficient, leading to numerical instability). Moreover, the model also fails to learn the diffusion-reace good accuracy. For the diffusiin-reaation equation, the MSE values are 2.5 x 10 1.7 x 10-1, 1, and 4.8 x 10-2 2 for train, in-dis-test, and out-dis-te dis- data, respectively. For the Allen-Cahn equation, the MSE values are 5.6 10- 2.5 10-1 and 4.3 x 10 for train, in-dis-test, and out-dis-test data, respectively. Even though the unknown equations do not seem too complicated, it is still difficult to solve them together with the PDE whole. The results show that for these particular problems, ANNs serve better because they allow better control during training, for example with constraints, and they produce more regularized outputs than high order polynomials. However, ANNs are not unique in their selection, but they are more convenient for our implementation.
(Fiuure 21b) and the Allen-Cahn (Figure 21c) equations with
A.9 ROBUSTNESS TEST FINN
In this section, we test the robustness of FINN when trained using noisy data. All the synthetic data is generated with the same parameters, only added with noise with the distribution. //oioonn///O... (0.0, 0.05). For the Burger's equation (Figure 2nd and Figure 23a). the average MSE values are 2.5 x 10 4.1 x 10 10--.. 2.4 x 10-3 6.6 10 and 2.5 x 10 = 4.0 x 10-6 for the train, in-dis-test and out-dis- test prediction, respectively. For the diffusion- sorption equation (Figure 22b and Figure 23b), the average MSE values are 2.5x 10:10-3440 44.5x 10-1 2.5 x 10 10--033-00000 333.7 x 10 and 2.5 x +3.7%
for the train, in-dis-test, and out-dis-test prediction, respectively. For the diffusion-reactionn equation
25

---

Under review as a conference paper at ICLR 2022
FINN (noise 0.05)
FINN (noise 0.05)
FINN (neise 0.05)
FINN (neise 0.05)
1.0
0.6
0.
0.5
0.5
04
0.5
0.
-0.5
-0.
0.0
0.0
0.0
0.
1.0
10000
t=] 1
0.1
0.5
0.0
0.0
-6.5
0
-0.5
Do73
Prodiction
Da7a
Prediction
De72
Peddction
D872
Prodiction
Burger's s
(b) Diffusion-sorption
PINN Inoise 0.05:
PINN Inpise 0. 0.05:
FINN (noise 0.05)
FINN (noise 0.05)
85
0.5
0.0
00
002
-0.5
05
-1.8
-]
-1.0
50.0
t= 10,
M
4.5
Redettam
Phedeciam
Peddecti
Reddetam
(c) Diffusion-reaction
(d) Allen-Cahn
Figure 22: Paired plots of the in-dis-test and out-dis-test prediction (left and right of the pairs, respectively) in the Burger's (top left), diffusion-sorption (top right), diffusion-reaction (bottom left), and Allen-Cahn (bottom right) equations obtained by training FINN with noisy data. The plots in the first row of the pairs show the solution over and (red line marks the transition from train to in-dis-test), and the plots in the second row of the pairs show the best model's solution distributed x.
Figure 22c and Figure 23c the average MSE values are 3.2 x -0-3 5.3 x 10 1.6 10 + 8.8 x 10 and 8.3 x 10- + 4.9 x 10 for the train, in-dis-test. and out-dis-test prediction, 22d]and and Figure 23d) the average MSE values are + 1.1 10-6 6, 2.5 10-3 3 + 9.1 x 10 and 2.5 10 + 6.3 x 10-6 for the train, in-dis-test, and out-dis-test prediction, respectively. These results show that even though FINN is trained with noisy data, it is still able to capture the essence of the equation and generalize well to different initial and boundary conditions. Additionally, the prediction is consistent, shown by the low values of the MSE standard deviation, as well as the very narrow confidence interval in the plots.
respectively. For the Allen-Cahn equation (Figure 22d
2.5 10
A. TRAINING PINN WITH FINER SPATIAL RESOLUTION
In order to determine whether the reduced accuracy of PINN in our experiments was caused by coarse spatial resolution (we only used 49, 26 and 49 x 49 spatial positions at Burger, diffusion- sorption, and diffusiin-reaationn respectively). another experiment was conducted per target equa- tion where the spatial resolution was increased to N₂ = 999, N 251, and N₂ = 99, N, = 99, respectively. As reported in Table 14 Figure 24 and Figure 25a the performance increased slightly but by far did not reach FINN's accuracy. Identical results were achieved the Burger's ' and diffusion-sorptioon equations but omitted due high conceptual similarity.
26

---

Under review as a conference paper at ICLR 2022
FIMN (noise = 0.05)
FINN (noise 0.05)
Dala Prediction
Dala Prediction
0.8
0.6
0.0
-0.2
0.2
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
FINN (noise 0.03)
FINN (noise 0.05)
Data
Data Preditioon
0.75
Prediction
0.6
0.50
0.5
0.55
0.4
0.00
0.3
-0.25
-1.00
-0.75
-0.50
-0.25 0.50
-0.75
(a) Burger's
(b) Diffusion-sorption
FINN (noise 0.05)
FINN (noise 0.05)
1.5
0.75
Prodiction
1.0
Prodiction
0.50
0.00
-0.5
-0.50
-1.0
-0.75
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.5
11.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
FINN (naisa
FINN (nalse 0.D5)
Data
1.5
Data
Prediction
Prediction
0.2
1.0
0.5
5'
0.0
-0.1
-0.2-
-1.0-
-033
-0.4
-1.5
-1.00
-0.75
-0'50
-0.25
0.00
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
(c) Diffusionnrreaciion
(d) Allen-Cahn
Figure 23: Prediction mean over ten different trained FINN (with 95% confidence interval) of the Burger's 's (top left), diffusion-sorption (top right), diffusion- (bottom left), and Allen-Cahe (bottom right) equations obtained by training FINN with noisy data for the in-di--test and out-dis- test (top and bottom, accordingly) prediction.
27

---

Under review as a conference paper at ICLR 2022
PINN (fine resolution)
PINN (fine resolution
PhyDNet (original architecture]
PhyDWet (ong architecture)
0.50
0.50
0.2
00
00
t- 50.
t- 50.
t- 50.
t- 10, y
0.25
9.5
0.2
-0.5
0.25
8. 3.0
-0.2
-1.
-0.5
Cata
Prodiction
Cots
Prodiction
Claba
Peodiction
Ciaba
Prodiction
In-dis-test prediction (left) and out-dis-test (right) (b) In-dis-test prediction (left) and out-dis-test (right)
Figure 24: Both left: plots of the diffusion-reaction equation's 's activator u using PINN trained with finer resolution dataset. Both right: plots of the diffusion-reaction equation's ' activator u using PhyDNet with the original network size.
Table 14: MSE of PINN trained using data with finer resolution from ten different training runs for the diffusion-reaction equation.
Table 15: MSE of PhyDNet using the original network size from ten different training runs for the diffusion-reaction equation.
Run
Train
In-dis-test
Oul-dis-test
Run
Train
In-dis-test
Out-dis-test
1.4x10-*
7,8x
9.3x 10
1,8x10-1
7.6 10 10
2
1.8x10-1
5.6x10-2
20-
6.5x10-2
3.5 x 10-2
3
8.6x10-3
1.5x
3
2.2x10-5
6.7x10-2
4.0 10-2
4
1.0x10-3
1,5x10-1
4.6x10
7.2x10-2
3.9 10
5
5.2x10-5
2.7x10-2
3.5x10-5
6.5x10-2
8.8 10
6
2.9x10-1
8.1x10-1
5.7x10-5
1.2x10-1
1.0 10
7
1.3x10
3.4x10-2
3.5x10-3
5.7x10-2
9.0 10
8
8.9x10-5
2.9x10-2
3.5x10
5.9x10-2
2.6 10
9
3.9x10-3
103
9
203
6.3x10-2
4.1 10
10
3.3x10
1,7x10
10
3.5x10
6.1x10-2
8.1x
A.11 PHYDNET WITH ORIGINAL AMOUNT OF PARAMETERS
To verify whether our reduction of parameters and the removal of the encoder and decoder layers caused PhyDNet to perform worse, we repeated the experiments for the three equations of interest using the original PhyDNet architecture proposed in Guen & Thome (202)). However, our results indicate no significant changes in performance, as reported in Table 15 Figure 24b] and Figure 25b Again, results for the inhibitor U2 well as for the Burger's and diffusion- reaction equations were omitted due to high conceptual similarity.
A.12 SOn. PARAMETERS AND SIMULATION DOMAINS FOR THE EXPERIMENTAL DATASET
Identical to Praditia et al. 2021I, the soil parameters and simulation (and experimental) domain used the real-world diffusion -soppioon experiment are summarized in Table 16 for core samples #1, #2, and #2B.
For all experiments, the core samples are subjected to constant contaminant concentration at the top Ud which can be treated Dirichlet boundary condition numerically. Notice that, for sample #2, we set u, to be slightly higher to compensate for the fact that there might be fractures at the top of core sample #2, that the contaminant break through the core sample faster.
28

---

Under review as a conference paper at ICLR 2022
PINN Mine resolution)
PhyDWet (original architecture)
Data
0.6
Data Preditionn
Prediction
0.5
0.4
0.0
00.5
0.0
-1.0
-1.5
-0.4
-2.0
-0.6
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.50
PINN (fine resolution)
PhyDNet (original architecture)
Data
0.4
Data Prediction
Prediction
0.2
8
-02
-1
-0.4
-0.6
-0.30
150
(a) PINN fine resolution.
(b) Phy original architecture
Figure 25: Prediction mean (with 95% confidence interval) of the activator (top) and inhibitor (bot- tom) in the diffusion-reaction equation at t 50 compared with the in-dis-test data.
For samples #1 and #2, is the flow rate of clean water at the bottom reservoir that determines the Cauchy boundary condition at the bottom of the core samples. For core sample #2B, note that the sample length is significantly longer than the other samples, and by the end of the experiment, no contaminant has broken through the core sample. Therefore, we assume the bottom boundary condition be no-flow Neumann boundary condition; see (Praditia al. 202T for details.
Table 16: Soil and experimental parameters of core samples #1, #2, and #2B. D the diffusion coefficient, is the porosity, Ps is the bulk density, L and r are the length and radius of the sample, lend is the simulation time, Q is the flow rate the bottom reservoir and u, is the total concentration of trichloroethylene in the sample.
Soil parameters
Parameter
Unit
Core #1
Core #2
Core #2B
D
m²/day
2.00 x 10-5
2.00 x 10-5
2.78
10-5
0.288
0.288
0.288
Ps kg/m²
1957
1957
1957
Simulation domain
Parameter Unit
Core #1
Core #2
Core #2B
0.0254
0.02604
0.105
m
0.02375
0.02375
N/A
Lend
days
38.81
48.88
Q
m /day
1.01 x 10
1.04 x 10
N/A
u.
kg/m²
1.4
1.6
29

---

Under review as a conference paper at ICLR 2022
Neural ODE runtime comparison
1.2
CPU
GPU
1.0
0.8
Runtime
www.
0.6
0.4
0.2
10000
20000
30000
40000
50000
60000
Number of batches
Figure 26: Runtime comparison of Neural ODE with single hidden layer consisting of 50 hidden nodes run on CPU and GPU for 000 time steps. The benefit using GPU starts to be seen when larger batch sizes (> 200) used.
13 ON TIME STEP ADAPTIVITY AND RUNTIME
We compare the runtime for each model, run on CPU with 19-9900K core, clock speed of 3.60 GHz, and 32 GB RAM. Additionally, we also perform the comparison of GPU runtime on a GTX 1060 (i.e. with 6GB VRAM). The results are summarized in Table 17 Note that the purpose of this comparison is not an optimized benchmark, but only to show that the runtime FINN comparable with the other model, especially when run in CPU. When run on GPU, however, FINN runs slightly slower. This caused by the fact that FINN's implementation is not yet optimized for GPU, More importantly, the Neural ODE package we use benefits only from larger batch size. As shown in Figure 26 GPU is faster for batch size larger than 20 000, whereas the maximum size that we use in the example is 401. With smaller batch size, CPU usage is faster.
In general, only the PINN model has a benefit when computed on the GPU, since the function is only called once all batches. This is different for all other models that have unroll prediction of the sequence into the future recurrently (except for TCN which is a convolution approach that faster on GPU). Accordingly, The overhead of copying the tensors GPU outweighs the GPU's parallelism benefit, compared to directly processing the sequence on the CPU iteratively. On the two-dimensional diffusion-reeacioon benchmark, the GPU's speed-up comes into play, since in here, the simulation domain is discretized into 49 x 49 = 2401 volumes (compared to 49 for the Burger's 's and Allen-Cahn, and 26 for the ceffusiion-sopttionn equations). Note that we have observed signif- icantly varying runtimes on different GPUs (i.e. up three seconds for FINN on Burger's 's on an RTX 3090), which might be caused by lack of support of certain packages for a particular hardware, but further investigation is required.
Additionally, we want to emphasize that the higher runtime of FINN on GPU not caused by the time step adaptivity. In fact, employing the adaptive time stepping strategy is cheaper than choosing all time steps to be small enough (to guarantee numerical stability). As we learn a PDE, we have exact knowledge to derive dedicated time integration scheme, but the adaptive Runge-Kute Kutta method is one of the best generic choices. As our PDE and its characteristics change during train- ing, time step adaptivity is a real asset, because for example the
(CFL.
30

---

Under review as a conference paper at ICLR 2022
Table 17: Comparison of runtime (in seconds) of single forward passes between different deep learning (above dashed line) and physics-aware neural network (below dashed line) methods on the different equations.
Eqn.
Model
CPU
GPU (GTX 1060)
TCN
0.423
0.130
ConvLSTM
0.052
0.079
DISTANA
0.059
0.098
Burger
PINN
0.033
0.007
PhyDNet
0.107
0.192
FINN
0.066
0.161
TCN
1.228
0.393
ConVLSTM
0.119
0.194
Diffusion-sorption
DISTANA
0.145
0.223
PINN
0.073
0.010
DNet
0.263
0.475
FINN
0.676
1.638
TCN
14.15
0.800
ConvLSTM
0.230
0.052
0.190
0.051
Diffusion-reaction
PINN
1.647
0.787
DNet
0.159
0.113
FINN
0.342
0.330
TCN
0.442
0.128
ComvLSTM
0.050
0.082
0.060
0.097
Allen-Cahn
0.033
0.007
PhyDNet
0.108
0.191
FINN
0.028
0.071
condition (Courant et al., 1967) would consistently change throughout the training. Therefore, the time step adaptivity is not a bottleneck, but rather solution for more efficient computation.
Furthermore, in relation to FINN's limitation (see|subsection A.2 topics like numerical stabilization schemes for larger time steps, adaptive spatial grid, optimized implementation in an High Perfor- mance Computing (HPC) setting, parallelization, etc. are still very interesting for future works.
31


Reviews:
[{"review_id": "tWoHtF6XqOG", "reviewer": "Reviewer_rFkv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\n\nThe paper provides significant improvement in performance in comparison to other models in a few orders of magnitude using 3-5 times less parameters than related models.\n\nThe results also suggest that modelling the spacial derivatives du / dx with the neural network directly results in better generalisation than modelling u(x,t and taking the derivative with respect to x (similar to PINN and other similar works). This is an interesting and counter-intuitive conclusion and worth of further exploration.\n\n**Weak points:**\n\nThe paper requires more explanation on the structure of the model. What is the input to phi_N, phi_A  and phi_D? How is R computed (it is not part of the equation in figure 1)? In Figure 1, why does phi_N signify both the first and the second spacial derivatives of u?\n\nIf the neural networks phi take only the values u, how are u(x,t) computed at the first time step? \n\nBased on figure 1, the right-hand side of the equation is a combination of four neural network outputs. Do you think it is equivalent to replacing phi_D phi_N - phi_A phi_N with a single neutral network taking all u_{i-1}, u_i, u_{I+1} (which essentially turns it into a Neural ODE model)?\n\nIt would be helpful to emphasise the differences between FINN, PINN and PhyDNet and other models in the methods section. Some of this explanation is provided in section 4.1, but it is worth emphasising and explaining more how the models differ, what exactly is modelled by a neural network and the motivation why FINN performs better than other models.\n\n\n**Questions:**\n\nPage 7: “PINN requires complete knowledge of the modelled system in form of the equation”. In this case, I am puzzled by a worse performance of PINN in figures 6 and 7 compared to FINN, if PINN closely follows the PDE that generated the data. Can you provide some insight why FINN performs better than PINN in this case? \n\nSimilarly, can you clarify why PINN requires more parameters than FINN (table 1)? Is it due to the fact that PINN also models the function u(x, t) or because of the bigger network?\n\n", "summary_of_the_paper": "The paper introduces Finite Volume neural network for modelling fluid dynamics inspired by the finite volume method. The paper models the velocity and the spacial derivatives as the neural networks. The work demonstrates more precise fluid simulation than the related physics-inspired models.", "summary_of_the_review": "The results in comparison to other models are compelling, particularly the model comparison on figures 6 and 7. However, the methods section of the paper needs more clarification and justification of the modelling choices in comparison to other papers. As the paper requires a significant re-write, I suggest a reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ab3WGs3GXoh", "reviewer": "Reviewer_LWqu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThis method can not only deal with smooth solutions of general PDE but also can deal with weak solutions of hyperbolic problems, which is a good point; It is also compared with many recent SOTA methods and is significantly ahead of recently published methods\n \nWeaknesses\n\nThe method limits the form of the equation, which will greatly increase the training complexity when it is extended to higher-dimensional problems\n\nThe problems studied by this method are relatively simple and are the fitting of some linear problems or low-order nonlinear problems. I am wondering if polynomial fitting will get better results.\n\nSome test examples can be supplemented. Such as equations containing an exponential function, the fitted coefficient containing singularity, or can this method be used for fluid equations, etc. \n\nThe robustness test of this method is absent. Can this method be used for noisy data?\n\nThe discussion of method limitation is absent.\n", "summary_of_the_paper": "This paper presents a compositional physics-aware neural network (FINN) for learning spatiotemporal advection-diffusion processes. It claims that the FINN outperforms pure machine learning and other state-of-the-art physics-aware models in all cases—often even by multiple orders of magnitude. However, the design of the network depends too much on the form of the equation, which leads to a very narrow application of the method.", "summary_of_the_review": "The demonstration of this method is pretty good, but there are too many restrictions on the problem. If the form of the equation is fixed, a simple polynomial fitting may achieve a better fitting effect than the neural networks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ZrtunbMNsg6", "reviewer": "Reviewer_TVaa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n1.\tThe authors propose very detailed experimental results, with both synthetic and real data\n2.\tThe background is provided in detail as well\n\nWeak points:\n1.\tThe authors claim in section 3 that convolution layers can only accommodate single types of boundary conditions. However, this is not correct since convolution layers at present can also handle cases I.e. periodic boundary conditions, this sentence may need rewriting\n2.\tIn figure 1, phi_N is shown to be both first and second-order derivatives, which may create confusion for the reader.\n3.\tThe training process is not completely clear, the authors describe the forward process, one would assume backpropagation of error for updating the neural network, but this is not explicit in the text. A figure demonstrating the whole training process, including the feedback loop might be helpful\n4.\tAt the end of section 3, the authors claim to use NODE in place of Euler for the reason of numerical stability, however, NODE also used Euler and does not mention anything about adaptive time-stepping, more details on this part would help to clear things up\n\n", "summary_of_the_paper": "The authors propose to model advection-diffusion partial differential equations as a composition of multiple neural networks. According to the authors, this leads to better generalization over different initial and boundary conditions for the physics system, and also the ability to learn different factors of the process as modeled by the terms in the PDE. Extensive experimental results are presented to support the author’s claims that their framework performs better than state-of-the-art.", "summary_of_the_review": "The paper has very thorough experimental results and good descriptions of the methods. However, the claims made in the paper may need a second look/ need some rewriting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X3UUAOSIRSb", "reviewer": "Reviewer_j4js", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very concrete paper with solid experiments. \nTo better assess this work, I would like to ask several questions:\n\n1. I find it misleading to call this model \"compositional neural network\", which usually refers to composing neural network as f \\cdot g = f(g). If I understand correctly, it is not what the paper does.\n\n2. If I understand correctly, the FINN model computes the derivative using the finite-difference method. It can be understood as a learn FDM/FVM stencil. I wonder how it compared to other learned FDM methods such as Machine learning–accelerated computational fluid dynamics by Brenner et. al.?\n\n3. If there is a spectrum of ML models vs conventional numerical solvers, I think this FINN method lies closer to the end of numerical solvers (which is nothing bad). Solver-like methods usually require fewer parameters and generalize better. Therefore, I think it could be valuable if the authors can add some numerical experiments comparing against the numerical solver. Are there any advantages (speed, accuracy, etc) to using the learn FVM vs the original FVM/FDM?\n\n4. Standard advection-diffusion equations are relatively easy to solve. I wonder if the idea proposed in this paper can be generalized and transferred to other PDEs?\n", "summary_of_the_paper": "In this work, the authors propose the finite volume neural network to solve advection-diffusion partial differential equations. The authors define a flux kernel as the sum of sub-kernel f_i on each element. And the authors define specific modules \\phi_D, \\phi_A, \\phi_N according to the form of the advection-diffusion equation. The model requires very few parameters to achieve the state of art results.", "summary_of_the_review": "I think the paper is above the threshold. If the authors can provide evidence that the proposed method outperforms (or has some relative advantage) compared to standard FVM solvers, I will be happy to raise the score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.  There is some disagreement about this paper, and several of the reviews are of relatively low confidence.  While I appreciate the effort that the authors have put into addressing the concerns of the reviewers, after going through the paper and the responses myself, I'm ultimately coming down on the side of the less positive reviews.  My reasoning, honestly, is that I think the authors are vastly overestimating the knowledge that the ICLR audience will have about numerical methods for PDE solutions.  Reading through the paper, I honestly have very little idea about how the actual numerical techniques are carried out, and it's unclear to me precisely where this method falls in between a traditional numerical solver an actual neural network.  Reading through the reviews, even the more positive ones, I don't think I'm alone in this perception (and the authors will hopefully believe me that these reviewers _are_ indeed emblematic of the subgroup of ICLR that is most experience with differential equations).  I really feel like either a substantial rewrite of the paper is needed, to make clear the full extent of the numerical methods being applied; or alternatively, the work may really be better suited for a numerical methods venue.
Decision: Reject