Neural Partial Differential Equations with Functional Convolution
Authors: Ziqian Wu
Xingzhe He
Michael Zhang
Yijun Li
Cheng Yang
Rui Liu
Shiying Xiong
Bo Zhu
Venue:  (2021)
Keywords: neural PDE
functional convolution
adjoint method

Abstract: We present a lightweighted neural PDE representation to discover the hidden structure and predict the solution of different nonlinear PDEs. Our key idea is to leverage the prior of ``"translational similarity" of numerical PDE differential operators to drastically reduce the scale of learning model and training data. We implemented three central network components, including a neural functional convolution operator, a Picard forward iterative procedure, and an adjoint backward gradient calculator. Our novel paradigm fully leverages the multifaceted priors that stem from the sparse and smooth nature of the physical PDE solution manifold and the various mature numerical techniques such as adjoint solver, linearization, and iterative procedure to accelerate the computation. We demonstrate the efficacy of our method by robustly discovering the model and accurately predicting the solutions of various types of PDEs with small-scale networks and training sets. We highlight that all the PDE examples we showed were trained with up to 8 data samples and within 325 network parameters. 

Under review as a conference paper at ICLR 2021
NEURAL PARTIAL DIFFERENTIAL EQUATIONS WITH FUNCTIONAL CONVOLUTION
Anonymous authors Paper under double-blind review
ABSTRACT
We present a lightweighted neural PDE representation to discover the hidden structure and predict the solution of different nonlinear PDEs. Our key idea is to leverage the prior of "translational similarity" of numerical PDE differential operators to drastically reduce the scale of learning model and training data. We implemented three central network components, including neural functional convolution operator, a Picard forward iterative procedure, and an adjoint backward gradient calculator. Our novel paradigm fully leverages the multifaceted priors that stem from the sparse and smooth nature of the physical PDE solution manifold and the various mature numerical techniques such as adjoint solver, linearization, and iterative procedure to accelerate the computation. We demonstrate the efficacy of our method by robustly discovering the model and accurately predicting the solutions of various types of PDEs with small-scale networks and training sets. We highlight that all the PDE examples we showed were trained with up to 8 data samples and within 325 network parameters.
INTRODUCTION
Problem definition We aim to devise a learning paradigm to solve the inverse PDE identification problem. By observing small data set in the PDE's solution space with an unknown form of equations, we want to generate effective neural representation that can precisely reconstruct the hidden structure of the target PDE system. This neural representation will further facilitate the prediction of the PDE solution with different boundary conditions. The right inset figure shows typical example of our target problem: by observing a small part (4 samples the figure) of the solution space of nonlinear PDE system F(z) b, without knowing its analytical equations, our neural representation will depict the hidden differential operators underpinning F (e.g., to represent the unknown differential operator V (1 + 2 V by training the model on the solution of V (1 (1+ (1==))
T=U-11xx111
Challenges to solve The nonlinearity and the curse of dimensionulity of the target PDE's solution manifold are the two main challenges for the design of high-performance neural discretization. An effective neural representation of PDE system plays essential role to solve these challenges. In retrospect, the design of neural PDE representations has been evolving from the raw, unstructured networks (e.g., by direct end-to-end data fitting) various structured ones with proper mathematical priors embedded, Examples include the residual based loss function (e.g., pwysics-informee networks
Raissi et al. 2020 Lu al. 2019 Raissi et al. 2019), learnable convolution kernels (e.g., PDE-Nets Long et al. 2018a| bl 2019) 9, and hybrid of numerical stencils and MLP layers (e.g., see Amos
Kolter] [017]] Pakravan al., 2020 Geng al. 2020 [Stevens & Colonius 2020). Following this line of research, we aim to devise lightweighted neural PDE representation that fuses the mathematical equation's essential structure, the numerical solvers' computational efficiency, and the neural networks" expressive power. In particular, we want to aggressively reduce the scale of both model parameters and training data to some extremal extent, while extending the scope of the targeted PDE systems to a broad range, encompassing equations that are both lincar and nonlinear, both steady-state and dynamic.

---

Under review as a conference paper at ICLR 2021
Translational similarity of differential operators Our neural PDE representation design is motivated by the historical successes of the various sparse, iterative numerical solvers in solving non- linear PDEs over the past decades. The key observation we have made is that the efficacy of a classical numerical PDE solver relies on the translational similarity of its discretized, local differential operators. Namely, the form of differential operator can be written as functional Czz, p) with respect to the the PDE unknown æ and the local position p. which is showed in the right inset figure. For example, for a linear Poisson system V Vx = b, C' is a constant function; for linear Poisson system with embedded boundary conditions, C' is a function of position p: for a nonlinear PDE V. PDE unknown æ (or both x and if it has embedded boundaries). For most numerical PDEs, these local functional operators can be parameterized and built on-the-ffy within the solver iterations. Such operators" locality further inspired the design of a variety of computationally efficient PDE solvers, among which the most famous one is the matrix-free scheme that has been used widely in solving large-scale physical systems on GPU. These local procedures for stencil creation have demonstrated their extreme performance in accommodating PDE solvers. From a machine learning perspective, these "translational similar" differential operators resemble the concept of convolution operators that function as the cornerstone to embed the "translational invariant" priors into neural networks ( see LeCun et al. 1995 1998).
(1 + +22) V = b, C a function of
Method overview In this work, leverage the PDE differential operators' "translational similarity" in a reverse manner by devising a local neural representation that can uncover and describe the global structure of the target PDE. At the heart of our approach lies in differential procedure to simultaneously describe the spatial coupling and the temporal evolution of local data point. Such procedure is implemented as a parameterized miero network, which is embedded in our iterative solving architecture, to learn the numerical process of converging from an initial guess to a final steady state for a PDE solution. We name these embedded micro networks "functional convolutions," for two reasons. First, fitting the parameters of these local embedded networks amounts to the exploration of the optimal function that best describes the observed solution of the unknown nonlinear PDE within functional space. Second, the local differential operators that span this functional space can be treated as numerically employing convolution kernels Hsiehh et al. (2018); Lin et al. (2013). Based on these functional convolutions, we are able to devise a learning framework by embedding the micro network architecture within an iterative procedure to 1) backwardly learn the underpinning, spatially varying structures of a nonlincar PDE system by iteratively applying the adjoint linear solvers and 2) forwardly predict the steady states of a PDE system by partially observing data samples of its equilibrium. We show that our model can simultaneously discover structures and predict solutions for different types of nonlinear PDEs. We particularly focus on solving elliptic boundary value problems that were less explored in the current literature.
MOTIVATING EXAMPLE: FORWARD NUMERICAL PDE
Naming convention We first show a motivating example to demonstrate the standard process of a forward numerical PDE solver. We take the simplest Poisson equation with Dirichlet boundary conditions as an example. The mathematical equation of a Poisson system can be written as V. Vx b for æ Ω, with as the PDE unknowns, b the right-hand side, and Ω as the problem's 's domain. The boundary conditions are enforced in a Dirichlet way (by assigning values directly) as æ & the domain boundary, with & the specified boundary values. To create discretized, numerical system to solve the equation, we use the symbol p to denote the position within the domain. The numerical solution of the PDE amounts to seeking an unknown function æ ap)) that can specify the value of æ in an arbitrary position within ?.
Linear PDE As shown in Figure we illustrate how to solve the Poisson system using a finite- difference method. We first subdivide the domain into cell (segment intervals in ID and squares in 2D) with the cell size of Ap. Taking the 2D case for example, we can derive the discretized Poisson equation by approximating the Laplacian operator each grid cell using the central finite difference method (-2;-1,) Ijj + 4zz, Ii.j-1 Σig-i) 11/ 1)/Ap" Ap² = beg,The The discretization of each cell forms row in the linear system, and the combination of all the rows (cells) forms a sparse
2

---

Under review as a conference paper at ICLR 2021
1D Poisson equation
2D Poisson equation
Xx-1
Xo,0
X1,0
Aa-100
X0
c(((pu))PG))
Xo,1
x111
1n-12
= -1,,-
Pn-1
((12))))=(112-)
-
118-1
0
0
Xao
0
30
No
X1
b2
X211
Dua
0
0
0
Xe-1
ba-1
A
X
b
A
X
b
Figure I: Schematic illustration to numerically solve the Poisson equation.
Alao, 0)s
A/s, Bizs
Alps, 012-
A(H Ups,
Converge to accuracy
Z0
21
222
Zn
Figure 2: Forward data flow neural Picard solver to predict the solutions of a non-linear PDE.
linear system Ax b solve. For a linear Poisson system, each row of A can be translated into a convolutional stencil instantiated with constant parameters, e.g., (-1,2,1) in ID and (-1,-1,4.-1.-I) in 2D. This convolution perspective can be used to accelerate the numerical solver by maintaining "conceptual"" A matrix without storing the redundant matrix element values in memory (we refer the readers to the matrix-free method Carey & Jiang (1986) for more details). This matrix-free nature indicates an extremely concise representation of numerical Poisson system (i.e., the matrix A) using a 1 x or x 3 convolution kernel with fixed values. We use the symbol to denote the convolutional representation of A. For a linear system, C' is independent from the values of and x.
Nonlinear PDE and Picard interation The nonlinear case of the Poisson system is a bit compli- cated. We can still use the matrix-free representation describe nonlinear Poisson system, but the parameters of this convolutional stencil now depends on both the local position p and the local unknown p(z). This dependency nonlinear and therefore we cannot find the solution by solving single Axr = b. Here we present an iterative scheme- the Picard method-to solve a numerical nonlinear PDE system. (see Picard 1893 Bai 2010) Let's consider the nonlinear Poisson equation as: V. (xx V x = b for E 52 and Diriclett boundary conditions on the boundary. The source of nonlinearity in this PDE includes the coefficient a<xx) which is dependent on the solution of æ., e.g.. a(x) 1 + æ². A simple and effective fixed-point procedure to discretize and solve the nonlinear Poisson equation, named Picard iteration, can sketched as follows:
while: [n xn-1 >
E xx = A (x.,)
with the matrix A representing the current discretized nonlinear Laplacian operator approximated by the value of unknowns from the previous iteration. The key idea is to employ linear approximation of a nonlinear problem and enforce such approximation iteratively to evolve to steady state (see Figure 2). To uncover the underlying structure of A. which can evolve both spatially and temporally, we make prior assumption that A can be described by kernel function C(æ(p), p). Such prior applies to most of the elliptic PDE systems where the spatial terms can be expressed as the combination of one several differential operators. From numerical perspective, C describes the local discretized interaction between an element and its neighbours. returns all the non-zero elements for each row i in A (think of A in matrix-free way).
It amounts to a function that
3

---

Under review as a conference paper at ICLR 2021
A A( 0)
Di
Fi+1
C(;; p, 0)
Solution
Intia
it
0
Functional Convolution
Ar
linearized Picard iteration
Figure Overview: To obtain the solution of A(x, 0)x = we first generate a random initial guess Then in each iteration step we apply the functional convolution C on to obtain the convolution kernels A A(x, 0). We obtain 1441 1 by solving A(x;, 0)x We repeat this process to obtain a converged solution.
METHOD: BACKWARD NEURAL PDE
Overview In this section, present neural PDE design motivated by the forward Picard solver with convolutional representation of the nonlinear system. Our framework consists of three key components: the neural representation of the convolution operator, the embedded Picard forward iterative solver, and the adjoint backward derivatives. The key idea is to differentiate the forward nonlinear Picard solver and build the neural representation for its sparse lincarization step. This differentiation is implemented by our functional convolution scheme on the linearization level and the adjoint Picard for the nonlinear iterations.
3.1 FUNCTIONAL CONVOLUTIONN
In conventional ML method, C' can be approximated by the combination of a set of kernels Long ct al. (2018a) or by solving deconvolution problem Xu et al.] (2014); Jin et al. (2017); Zhang al. 2017p. However, these strategies do not suit scenario, where the instant linear effects of the system should be approximated by extracting the nonlinear effects of C. A natural choice to approximate this spatially-and-temporrd takes the form of C(x(p), p, 0), with 0 as the network parameters. Numerically, the global matrix A can be fully parameterized by C(x(p). p. B) by assuming the fact that C is non-weight-she whight--saadd convolution operator in the spatial domain. As illustrated in Figure [ such neural network can be further incorporated into conventional nonlinear Picard solver to obtain the forward steady-state solution by solving the linear system A(æ,, f1 = b(æx), where black-box sparse linear solver can be used as conventional simulation program. The formal definition of functional convolution written in the kernel way is
varying kernel function is devise a neural network, which
A(x(p)).0)
Σ [Cla(W)))) P.EN(p))
where (ax(p,)) 8) is the 7th row of matrix A, (p:)) the neighboring positions of pts aNN(p.) is all the neighboring elements (all channels) around the position p; and e(p;) is the unit vector representing the jth column in A.
To specify the 2D example, equation 2 has the following form
A A(xm.n),
Σ Σ
1=-15=-1
where Zm., is the element that lies in row m, column n of the feature map. The input N (Z = (Im ji for i -1,0, is the flatten vector of neighboring elements of pixel Tmaa After
4

---

Under review as a conference paper at ICLR 2021
simple neural network C' with parameters 8, obtain the output C(N 0), which is a vector with the same length of N (Im,))
3.2 ADJOINT DERIVATIVES
To train our functional convolution operator in an end-to-end fashion, we calculate the derivatives of the loss function L regarding the network parameters 0 following the chain rule. For neural PDE with n layers for the outer Picard loop (see Figure B we can calculate the sensitivity aL/ as:
aL aL dx
0L dxn d +
ØL da
Oxz dx
+
(4)
00 Ox 00 Dx dxn-1 00
Dx 8xn-1
00
For layer that maps from a; + #i41 by solving A(x,, 0) = b, we can express its backward derivative y: as
da -A (x, 0)
(A
yi dx
Di+1,
(5)
Ox;
which can be solved by two linear systems, including forward equation and a backward adjoint equation:
A 0)
(6a)
0A(x;, 0) dæ: 11-1
A 0)y.
(6b)
Similarly, we can get 0L/00 as:
da,+1 - -
0A 2x 0) Ox: xi+1
2; 00
(x, 0)
(7)
which can be calculated by solving additional adjoint linear system:
0A(x; 0)
A 0)zi
00 Ti+1
(8)
To calculate the terms 0A/Ox; and 0A/30 in the chain, take advantage of the sparse nature of A by calculat- ing aC/Ox; and ac/00 first and then distribute their values into the non-zero entries of the global matrix. Because C(æ(p), P. 0) is functional convolution operator represented by standard neural network, its derivatives regarding the first- layer imput æ and the parameters can be calculated in straightforward way by the auto-differentiation mechanism. The overall algorithm is summarized as in Al- gorithm
Algorithm Backward derivative nonlinear PDE boundary value problem
Input: æ1 b, C, T.
Output: 0T./00
//Forward linearization: for = 0 + N - do
Solve A 0)x,+1 b:
end for //Backward adjoint: do
for i N
Solve adjoints (bb) and for 0m:41/00
0L/00 + (ÖL 0 (10++
end for
NETWORK ARCHITECTURE AND DATA TRAINING
The network we use in this study summarized in Table 2 We use hybrid method of IpOpt Wächter (2009) and Adam optimizer to optimize the parameters int the neural networks. Despite its fast converging rate, IpOpt optimizer typically converges to a local minimum or fail to converge due to a bad initialization. By introducing the hybrid optimization method, we are able to solve the problem. The optimized parameters from IpOpt are then oscillated and refined by Adam optimizer for certain number of iterations. In case that the loss not converge, then use the optimized parameters from Adam as the initial guess to warm-start the IpOpt and repeat this IpOpt-Adam process until the loss converges the tolerance. For the Adam optimizer, the parameters
5

---

Under review as a conference paper at ICLR 2021
be learning rate 1c - 3, 31 0.9, B2 0.999, c 1c- 8. We compare the performance of our hybrid optimization method with Adam and a typical SGDM method in solving the ID Poisson problem in Figure 6 The results show that the hybrid method not only obtains faster converging rate, but also converges to an extremely small loss compared with other methods.
5 RESULTS
We test the efficacy of our method by using it to fit a set of PDEs, ranging from the standard linear Poisson's 's problem to highly nonlinear temporally evolving PDEs. We build our neural network architecture by implementing both the differentiable Picard iterations and a simple fully-connected network C++ 17. The training process for all the examples was run on an i7-8665 four-core desktop. The PDE and boundary conditions, sample size, and test results were summarized in Table in Appendix.
5.1 ID EXAMPLES
Constant kernels We first test our framework by fitting the underlying structure a standard 1D Poisson problem V Vx with Dirichlet boundary conditions. Here the target true solution is set to be x = (ap)?. with p as the spatial coordinate of the data sample and a as a coefficient that varies for different data sample. The training dataset consists of four sample points, observed from the two solutions of the equations parameterized with different values of a and boundary conditions. The 4-size training data is sampled on 128 x grid with two layers of picard network. After training, we run 16 PDE tests with varying a and boundary conditions and obtain an averaged MSE loss as 8.3e-20. The predicted solutions all 16 test cases match the analytical curves with the maximum MSE 1.5e- 14, as shown in in Figure 9a
Next, we test the same framework on another two Poisson problems. The first problem is Vx with Neumann boundary conditions which does not have an analytical solution (see Figure 9b). The prediction results from our model are shown in Figure 9b with an averaged MSE 5.0e-29. The other problem is with the target solution as x = sin(ap). The results are shown in Figure with the 9.8e-4 MSE loss compared with the true solution.
Stability test with noisy input We further conduct stability test by adding noise values to the setting of the ID constant kernel experiments. The training dataset consists of two sample points, observed from the two noisy solutions of the equations. We test our framework with various scales of noise from .1] to |-.35, .355 with step size of .05 with respect to the extreme values in the target solution. The training data is sampled on 32 x grid with two layers of picard network. We compare our framework with denoised solutions in Figure T0 The figure shows the framework can automatically obtain precise solution even though our framework cannot access accurate solutions during the training procedure.
Spatially varying coefficients Next we test our model by predicting the solutions of Poisson PDE with spatially varying coefficients. The PDE has the analytical formula V (1+ |=p]) |1+1|Fp)) Vx with p as the spatial coordinate of the data sample. We build the training dataset by solving the equation with randomly set boundary conditions on a 32 x grid. The data sample size is and each data sample has the full observation of the solution space. The input of the network is the current values of æ and the sample positions p. We show that our model can precisely uncover the distribution of the hidden latent space behind the Laplacian operator with spatially varying coefficients by fitting a functional convolution operator that predicts the solutions of the PDE in the forward process (see Figure 3. The average MSE loss is 1.3e-06 for this case.
Non-linear equations In this example, we demonstrate the ability of our neural PDE solver by solving a non-linear PDE with the form: V (1 |a| + sin((x] * .001)) = Dirichlet boundary conditions are set on the two endpoints of the domain. The training dataset is generated by solving the PDE with standard Picard iterations. The number of the neural PDE network layers is set to be 32 1 discretization for training. As shown in Figure [I]
5. We employ 4 solution samples our model can precisely uncover the hidden nonlinear structures of the PDE kernel and predicts the numerical solution by employing the leameed functional convolution through the Picard iterations.

---

Under review as a conference paper at ICLR 2021
5.2 2D EXAMPLES
Poisson equation We then expand our model to solve 2D problems. Similarly, we start from employing our model to predict the solutions for two standard 2D Poisson problems V Vx= b with Dirichlet boundary conditions, whose target true solutions are set to be æ (apx)) و + op, and a: = sin(apa + app)) respectively. Here Pa and Pr refer to the x axis and y axis coordinates of the data sample. We use 4-size data samples, which are sampled on a 32*32 grid, to train the neural network in both cases. To evaluate, we run 16 test cases for both the two problems and the results are shown in Figure E We obtain an averaged MSE loss at 5,4e- 14 for the first problem and 3.7e-3 for the second problem.
Helmholtz equation In this example, we test our model's performance by predicting the solution of Helmholtz equation V Vx = O. We set two different types of varying Dirichlet boundary conditions for this problem, one as æ = -a/ +p²) and another as æ = a sin (0mw(00.22)) (0.02pp. +..02p.)) with varies across the data. We use 4 data sample with respect to two types of boundary conditions with varying coefficients to train the neural network. For each type of boundary conditions, the training data is sampled on a 32 x 32 grid in two different domains respectively. The results are exhibited in Figure S In predicting solution, we achieve an averaged MSE of 5,25e-27 and 9.3e-29 in the two specific domains of the first type and For 3.5e-25 and 3.9e- 18 for the second type.
Wave equation In this example, we demonstrate the ability of our Neural PDE solver by solving a time-dependent wave equation: V Vx Fe We use a standard finite difference method to generate the training data, which is 6-size data sample with each training sample indicating the target solution at the th time step (1 < n < 6). Our model is trained to map æ from the n - frame to the n frame. The training data is sampled on a 49 49 grid with a source of æ = sin s6n(60(n * df)) at the center of the grid, where dt is the time interval. With this observation of previous frames of a time-related wave function, our model is able to predict the following frames. The model's performance is tested by predicting the following 42 frames after the first 6 training frames. The training data and the predicting results are showed in Figure 13 and Figure 14 With an average MSE loss of 6.9e-4, we show that our model can precisely uncover the intrinsic structures of the kernel with sparse observation and can predict the numerical solution of it in a period of time.
Navier-Stokes equation equation:
We further demonstrate our model's ability in solving the Navier-Stokes
OE Ot
+ + + VP=
V E=
where, J stands for the velocity of the fluid, p indicates the pressure and f the body force. In each time step, model is trained to accomplish the projection step which has form of Poisson equation through the finite difference discretization method. The 6-size training data is sampled on a 32 x 32 grid with fluid source in the left bottom corner of the domain, and the model is tested for 50 frames. The training data and the predicting results are showed in Figur [2] With an averaged MSE loss of 4. -50 we show that our model can precisely uncover the intrinsic structures of the projection step in solving the Navier-stokes equation with sparse observation.
5.3 COMPARISON WITH OTHER METHODS
We compare our framework with CNN baseline and PINN Lu et al. (2019).
Comparison with CNN baseline We evaluate functional convolution model by comparing its performance with other naive convolutional neural network (CNN) structures in solving two typical problems targeted in this study: 1) ID Poisson problem and 2) 2D time-dependent wave equation. To solve the 1D Poisson equation, we set up the CNN baseline structure as a 5-layer network consisting of three ID convolution layers with a ReLU layer in between each two. The 2D time-dependent wave equation is specified by Equation V Vx áx² The CNN baseline structure is a 5-layer network, which is described in Table I Figure 7 shows the results. The figure shows that our framework converges fast and reduce the loss dramatically compared with the baseline. The details of the
experiment could be found in Section B.1 in Appendix.
7

---

Under review as a conference paper at ICLR 2021
(2)
Figure 4: Test cases of ID and 2D Poisson equations. (a)Test cases for predicting sine function.
(d)Test cases for predicting sine function with 10% noise. (b-f)Test cases in predicting the 2D) Poisson equation with analytical solution (b,c) sin(a Pa + a po), (e.f) (ap.) s app
(e]
(9)
Figure 5: Test cases predict the solution of 2D Helmholtz equation with different boundary conditions in different domains. Right figures show the top view of the solution.
(209)) in the setting of Helmholtz equation system. Specifically, the comparison is conducted with the Helmholtz equation V O. The comparison framework and PINN are trained and tested on 32 x 32 grid. Figure & shows the prediction results for PINN and our framework. Our framework achieves MSE of 6.05e 15, while PINN achieves MSE of 1.66e 6. The details of this experiment could be found in Section B.2 in Appendix.
Comparison with PINN We compare framework with PINN
the Dirichlet boundary of
(p + p²). Both our
shows
RELATED WORKS
PDE networks
Long et al. (20 8a
2019) explore using neural networks to solve the Partial
differential equations (PDEs). & Shi 2017 formulate the ResNet as a control problem of PDEs on
manifold. Raissi et al. (019) embeds physical priors in neural networks to solve the nonlinear PDEs. Han et al. (2018) handles general high-dimensional parabolic PDEs, Brunton et al, (2020) gives an overview of neural networks in turbulence applications. Wu et al. (022) train generative adversarial
8

---

Under review as a conference paper at ICLR 2021
networks to model chaotic dynamical systems. Han al. (2016) solves high-dimensional stochastic control problems based on Monte-Carlo sampling. Raissi 2018) approximate a deterministic function of time and space by deep neural network for backward stochastic differential equations with random terminal condition to be satisfied. Wang et al. (2019) first propose to use reinforcement learning (RL) aid the process of solving the conservation laws. Jagtap et al. (2020b) propose a neural network on discrete domains for nonlinear conservation laws. Jagtap et al. (2020a) employ adaptive activation functions to solve PDEs and approximate smooth and discontinuous functions, Pang et al. (2019)
combines neural networks and Gaussian process to solve PDEs.
Prior-embedded neural simulators Many recent learning physics works are based on building networks to describe interactions among objects components (see [Battagiia et al. (2018) for survey). The pioneering works done by [Battaglia et al. (2016) and Chang et al. (2017) predict different particle dynamics such as N-body by learning the pairwise interactions. Following this, the interaction networks are enhanced by graph network by Kipt et al. (017)) for different applications. Specialized hierarchical representations by Mrowea al. (2118). residual corrections by (ajay et al. (2018). propagation mechanisms by Li et al. (2019), linear transitions by Li et al.] (2020) were employed to reason various physical systems. On another front, modeling neural networks under dynamic system's s perspective has drawn increasingly attention. In 2018 Chen et al. (2018) solves the neural networks as ordinary differential equations (ODEs).
7 CONCLUSION
In this paper, we introduced neural PDE, a machine learning approach to learn the intrinsic properties of PDEs by learning and employing novel functional convolution and adjoint equations to enable the end-to-end training. Our model resents strong robustness against the arbitrary noise. The main limitation of our work is that we assume the PDE systems are sparse. That is, the relation is restricted locally. To enable the constrain in larger area, one can enlarge the kernel size, but this can potentially cost much more computation and memory. For the future applications, we plan to apply the method to solve real-world 3D applications, such as incompressible fluids and nonlinear soft materials. We also plan to scale up the system by leveraging the sparse iterative linear solver for solving the linearized PDEs in each iteration.
REFERENCES
Anurag Ajay. Jiajun Wu, Nima Fazeli, Maria Bauza, Leslie Kaelbling, Joshua B Tenenbaum, and Alberto Rodriguez. Augmenting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing. In 2018 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3066-3073. IEEE, 2018.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as layer in neural networks. arXiv preprint arXiv: 000330043, 2017.
Xiaoli Bai. Modified Chebyshev- iteration methods for solution of initial value and boundary value problems. Texas A&M University, 2010.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4510, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:180.001266 2018.
Steven I. Brunton, Bernd R Noack, and Petros Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477-508, 2020.
Graham F Carey and Bo-Nan Jiang. beenent-by-elmment linear and nonlinear solution schemes.
Communications in Applied Numerical Methods, 2(2):145-13, 1986.

---

Under review as a conference paper at ICLR 2021
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. International Conference on Learning Representations, 2017.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. Coercing machine learning to output physically accurate results. Journal of Computational Physics, 406: 109099, 2020.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34)885058811 2018.
Jiequn Han et al. Deep learning approximation for stochastic control problems. arXiv preprint 1xXiv:161.00722 2016.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learn- ing neural pde solvers with convergence guarantees. International Conference on Learning Representations (ICLR), 2018.
Ameya D. Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physicx, 404: 136, 2020a.
Ameya D. Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physies-informed neural networks on discrete domains for conservation laws: Applications forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365: 13028, 2020b.
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9): 4509 4522, 2017.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. International Conference on Learning Representations, 2017.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 336) 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1:22)222223333 1998.
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake, Prop- agation networks for model-based control under partial observation. In International Conference on Robotics and Automation (ICRA), pp. 1205 1211. IEEE, 2019.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman operators for model-based control. In International Conference on Learning Representations (ICLR), 2020.
Zhen Li and Zuoqiang Shi. Deep residual learning and pdes on manifold. arXiv preprint arXiv:1708.051 2017.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400. 2013.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning (ICML.) pp. 3208-3216, 2018a.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216, 2018b.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 399: 10892, 2019.
10

---

Under review as a conference paper at ICLR 2021
Lu Lu, Xuhui Meng, Zhiping Mao, and George E Karniadakis. Deepxde: A deep learning library for solving differential equations. arXiv preprint vrXiv:197..05022 2019.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and
Daniel L Yamins. Flexible neural representation for physics prediction. In Advances in neural information processing systems, pp. 8799-8810, 2018.
Samira Pakravan, Pouria A. Mistani, Miguel Angel Aragon-Calvo, and Frederic Gibou, Solving inverse-per -ederse-peepieee problems with physics-aware neural networks, 2020.
Guofei Pang. Liu Yang, and George Em Karniadakis. Neural-net-induced gaussian process regression for function approximation and pde solution. Journal of Computational Physics, 384:270 288, 2019.
Emile Picard. Sur I' I'application des méthodes d' appprxxmmations successives l'étude de certaines équations différentielles ordinaires. Journal de Mathématiques Pures et Appliquées, 9:217-272, 1893.
Maziar Raissi. Forward barkwad stochastic neural networks: Deep leamnin of high-dimensionn partial differential equations. arXiv preprint arXiv:1804.070 07010, 2018.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informet neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, (67((111)) (004611):126-10880 1030, 2020.
Ben Stevens and Tim Colonius. Finitenet: A fully convolutional Istm network architecture for time-dependent partial differential equations. arXiv preprint (.Xxx:2002.03014, 2020.
Andreas Wächter. Short tutorial: getting started with ipopt in 90 minutes. In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zenttee für Informatik, 2009.
Yufei Wang, Ziju Shen, Zichao Long, and Bin Dong. Learning to discretize: Solving 1d scalar conservation laws via deep reinforcement learning, 2019.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Heng Xiao, et al. Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics, 406: 109209, 2020.
Li Xu, Jimmy Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvo- lution. In Advances in neural information processing systems, pp. 1790-1798, 2014.
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang, Learning deep emn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3929-3933, 2017.
A ABLATION TESTS
The neural networks are trained by hybrid method of IpOpt and Adam optimizer. Here we demonstrate our hybrid optimizer's capability to outperform typical Adam optimizer and typical SGDM optimizer, which is shown Figure
B COMPARISON WITH OTHER METHODS
B.1 COMPARISON WITH NAIVE CNN STRUCTURE
We evaluate functional convolution model by comparing its performance with other naive convolutional neural network structures in solving two typical problems targeted this study. For the first solve the 1D Poisson equation, we up the baseline structure 5-layer network
11

---

Under review as a conference paper at ICLR 2021
100 (0000
-10
82
120
200
100
150
200
Boration over ondic datasot
Icradoos over entire dotasct
Figure 6: Training our model to predict the solution for the 1D Poisson problem V. Va = O using different optimization methods. The IpOpt+Adam optimizer stops at early converge with the total loss of 8.5e-25.
consisting of three convolution layers with ReLU layer in between each tWO.
The second The baseline
problem is the 2D time-dependent wave equation specified by Equation V V be a 5-layer network, which is described in Table For each of the
2pz
for this problem also baselines, the input is set to be the right hand side value and the output is the predicted solution. The results are shown in Figure 7 which demonstrates that our functional convolution model outperforms the naive convolutional neural network both in accuracy and converging rate.
25
OHAGE
Mackd
(a)
(b)
Figure 7: comparison between predictions from Naive CNN networks and our model to predict solution for (a) the ID Poisson problem V Væ 0 and (b) the time-dependent wave equation.
B.2 COMPARISON WITH PINN
We compare our framework with PINN Lu et al. (2019) the setting of Helmholtz equation system. Both models are trained and tested in 32 32 grid. Figure 8 shows the prediction results for PINN and our framework.
C ID EXAMPLES
C.I ID POISSON EXAMPLE
We employ functional convolutional model with the aim to solve typical 1D Poisson problems with constant kernels, the testing results of which are shown in Figure 9
12

---

Under review as a conference paper at ICLR 2021
40
31
(6)
Figure 8: Comparison with PINN Figure (a) shows the predicted value of PINN with MSE of 1.66e-6.
Figure (b) shows the predicted value of our framework with MSE of 6.05e-15.
(a)
(b)
Figure 9: Two test cases for predicting the solution of 1D Poisson equation V Væ b with (a) target solution & (ap)² and (b) = 0. The dots represent the predicted values and the curves indicate the true values.
C.2 1D STABILITY TEST WITH NOISY INPUT
We conduct stability test by adding noise values the setting of the ID constant kernel experiments. We test our framework with various scales of noise from [-.1,.1] to [-.35, .35] with step size of .05 with respect to the extreme values in the target solution. We compare
framework with denoised solutions in Figure IO The figure shows the framework can automatically obtain precise solution even though our framework cannot access accurate solutions during the training procedure.
C.3 ID SPATIAL VARYING AND NONLINEAR EXAMPLE
We also train the model to predict the solution of V (1 + (Tp)) = and V 7 (1 + |a|| + sin([a) Vo = The results are shown in Figure II
C.4 2D NAVIER STOKES EXAMPLE
The 6-size training data sampled on 32 x 32 grid with fluid source in the left bottom corner of the domain, and the model is tested for 50 frames. The functional convolution network is set to be 3 x absolute error between the two of Frame 25, and 50 in Figure| T2
x 6 x x 3. We show the ground truth by a typical fluid solver, our prediction, and the
13

---

Under review as a conference paper at ICLR 2021
testing Exmppee (nateen8.15)
Seatng sumple (nose-0.30
gruud tullnal
-00
-10
-1.5
(4)
(b)
0.026
0A2
007
-1.
0
0.15
0.25
sampling points
noke
Figure 10: Figure (a) and (b) show two testing cases with noise of and with respect to the extreme values in the same target solutions. The MSE errors of the 16 testing cases for them are 0.0005 and 0.0274. Figure (c) shows one training sample with noise. Figure (d) shows the relationship between the noise in the observation and testing MSE.
Figure II: Test cases for predicting the numerical solution of an unknown Poisson PDE. The left figure shows function domain of all tests to predict the Poisson PDE with with spatially varying coefficients. The right figure shows tests to predict the solution of nonlinear PDE specified by V |2| + sin((ar) =001)) 0.
D WAVE EQUATION
Figure [3] shows the training data of the wave equation. Figure 14 shows the predicted solution of time dependent wave equation.
14

---

Under review as a conference paper at ICLR 2021
frame=1
frame 225
frame 50
Figure 12: 2D Navier Stokes example shows the ground truth by typical fluid solver, our prediction, and the absolute error.
framee3
framee5
0.8
04
22
02
04
85
0R
87
28
22
15
02
32
82
Cal
(b)
(d
Figure 13: The training data to solve the time-dependent wave equation specified by Equation V Vv 8= 94 (a-c) The target solution at timestep 1, 3, 5, with top view and 3D view. The equation solved in domain = [0, 1] x [0, 1], here we only show the plots in [0.2, 0.8] x [0.2, 0.8).
E DETAILS ON NEURAL NETWORK STRUCTURES
We show the details of the naive CNN structures in Table [ which are trained baseline compared to our convolution model. We also refer readers Table 2 for the details of neural network structure and training parameters across different examples.
15

---

Under review as a conference paper at ICLR 2021
-.0
1d
[d]
Toune
Figure 14: The predicted solution the dime-dependent wave equation. (a-h) The predicted solution at timestep 6, 12, 18, 24, 30, 36, 42, 48 with top view and 3D view. The equation is solved in domain [0, 1] x [0, 1]. here we only show the plots in [0.2, 0.8) x [0.2, 0.8).
Table 1: Naive CNN baselines
Baseline
neural network structure
grid
training
smaples
ID CNN structure
enn fmm.odin=s..ouss-.eeme-3333 ReLU, amm/h=sl,.ot=:keeeee33)j.... ammlddin=1..nmt11emmeel3)) cml.d/in4..o00222kemm11--3 ReLU, uml(i=22ooue2Aeeml333)RRLLLU omllddin=22ouuss1eeeee))))
16
2D CNN structure
20
16

---

Under review as a conference paper at ICLR 2021
Table 2: Neural network structure and training parameters
Equation
Dimension
Network
Parameters
# Test sample
Test MSE
Training sample
V
Va
b with target solution
128 x
3
4 x
x
31
16
8.3e:20
(ap)
16
5.00-29
Væ
b with target solution
32
3x 3
31
16
9.8c-4
(ap)
with solution
3x4x4x3
31
16
from 5e-4 to 2.76-2
8 (ap) with noise
V1
V=
x
6X6x6X6X6X3
108
16
1.3e-6
2
3x6x6x3
45
16
2.1e-5
001)
Vx =
target solution
32
3x14x14x14x
(ap.)" op?
14x3
V. Vie = b with target solution
32.x 32
3X10x10x1UX
293
16
3.7e-3
(ap, + ap.)
x0
V. v
with boundary
32
3X3x5x5x5X3
68
16
5,36-27
== a/( - )
with bound- V V (0.((0.02p..
32
3x5x5x5xXx3
16
9.36-29
0.02p.])
Ve
Dr2
e²g with center
49 x 49
3x5x5x5xxxx
68
42
6.9c
source 660( d()) ZVF PAd Vp
32
3x6x6x6x6xx
87
50
4.09e-5
17


Reviews:
[{"review_id": "tYxwypDbtUc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the work proposes to use neural networks to learn a kernel C(x,p) for PDEs. It embeds the neural network into iterative solvers and trains it with the Adjoint method.\n\nThe writing is clear in general but notation-heavy. For example, it could be better to define notations such as $A$ before using it. \n\nStrong points: \n- Clear, \n- Well-motivated,\n- Theoretically solid, \n- The example given on page 3 is quite helpful.\n\nConcerns:\n- The idea seems similar to the previous works.\n- Lack of comparison and benchmarking.\n\nI am not very certain about evaluating the novelty of this work. The idea to approximate the kernel with the neural networks has been proposed and studied in (https://arxiv.org/abs/2003.03485, https://arxiv.org/abs/2010.08895). The major difference seems to be that they directly learn solution operators, while in this work we embed it into iterative solvers. \nIt will be great if the authors can help me understand the contribution beyond the previous works.\n\nAnother concern is about the experiments. The test equations (Poisson, Helmholtz, Wave Equation) presented in the paper seems fairly simple. I wonder how other methods such as PINN, or the numerical solver perform on these equations. It will be great to have some benchmarks and comparisons with existing works. It can help me better evaluate the performance of the method.\n\nAlso, I found the scale for the neural network is very small. The authors claim larger networks are not needed, but it's better to have some justification or experiments.\n\nQuestions:\n1. how does it differ from the existing work?\n2. how does it empirically compare with numerical solvers or other deep learning-based methods?\n3. is it possible to try large networks in the experiments?\n\nRecommendation:\nIn general, I found this work interesting and concrete, but I am not certain about the novelty. Therefore, I would like to put this paper on margin.\n\n---\n_Updated review:_\n\n>The updated manuscript has some substantial improvements.\n>\n> I feel the biggest problem is that the authors didn't clearly state the problem settings. If I understand correctly, in their framework the equation is fixed but unknown. The training data are several points in the domain (with parameters input) and testing data are other points. So basically, we doing interpolations. But even the PDE is unknown, they do assume some structure of the PDE, I think.\n>\n> Other PDEs frameworks are either 1. solver-type: the equation is known and fixed, they directly solve for the solutions. 2. operator-type, the equations are unknown and changing. Train on inputs-outputs for several equations, and test on others. Their setting is quite different. I guess it's the reason their performance is much better in the updated comparison. On the other hand, it's also hard to evaluate their performance since there are no fair benchmarks.\n\n> In general, I feel this paper is novel and concrete, while it's not very complete and well-presented. I agree with other reviewers that this paper is not ready to publish.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dYuAViybQ5t", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "ICLR Neural PDEs\n\nSummary: \n\nThis paper aims to use neural networks to find hidden structure in PDEs and predict their solution. Their neural network examples are extremely small (up to 325 parameters) and require little data (up to 8 samples). Instead of a classical convolution, which combines the neighbors with fixed weights, they learn a \"functional convolution\" that combines the neighbors with weights that are themselves functions of the neighbors. This takes advantage of \"translational symmetry\" in discretized differential operators. The tiny network is then embedded in a Picard forward iterative procedure. An adjoint backward gradient calculator relies on being able to do auto-differentiation in neural networks. They specifically consider elliptic boundary value problems. It is key that the PDE systems are sparse (dependent on a limited number of neighbors).\n\nStrong points:\n\nI am very impressed by the small networks and limited training data. \nSome of the test MSEs are quite impressive. \nTo my (albeit imperfect) knowledge of the literature on machine learning for PDEs, this is quite creative. It also seems like a big step forward as far as low error with limited data.\nIt's nice to see the variety in the six test cases.\n\nWeak points/Clarification questions:\n\nThis paper needs more details on the training and testing data. \n- What counts as one data sample? \n- The input & output is not always clear.\n- How much variance is there within the training & test data? For example, for the \"constant kernels\" case, I can tell from Figure 8. However, for the \"spatially varying coefficients\" case, I think we never see the data. Also, for some cases, I don't think it's mentioned what is varied between examples.\n- It's sometimes unclear from the captions whether we're seeing training data or test data, or whether it's the true data or the results.\n- I'd like to see both training & test error to see if there is overfitting.\n- I don't see any mention of validation data, which is often what people use to choose hyperparameters. What did you use to choose hyperparameters? If the test data was used for this, we need new test examples, as test examples need to be held out until it's time to report errors. \n\nI may have missed this, but about how long does it take a train your method, and how long does it take to use the trained network for prediction? \n\nThe text in the figures are often too small to see. All parts of Figure 3 (a) & (d) are difficult to see. \nThe Related Work section (Section 5) lists a lot of papers, but doesn't explain what advantages your paper has.\n\n~~~~~\nUpdate:\n\nI think that the revised paper is an improvement,  but it's not ready. I think there is still missing information to make it clear what you did (as the other reviewers have commented as well.) I am particularly concerned that we still don't have a comparison of train & test errors for each network, and that we don't still know which dataset was used for selecting hyperparameters (train vs. val vs. test). These are crucial questions for deep learning results, and I always check this when I peer-review. I reduced my rating based on these concerns. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FhVjYOZu-bp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a neural network based solver for PDEs based on the Picard Iteration. In the numerical experiments section the paper applies the method to solve 1d or 2d PDEs.\n\nDisclaimer:\nI am not an expert in PDE solvers but I am researching topics related to your frequently cited works of Kipf et. al., Battaglia et. al. & Raissi et. al.. Therefore, my knowledge on PDE solver might be a limiting factor to understand the complete work.\n\nReview:\nFor me personally it is really hard to parse the paper and get a rough understanding of the paper. I cannot fully understand the application, why it should be better to numerical solvers, how it roughly works, what kind of training data is used. By now I have reconstructed the genreal idea from the details presented but I am uncertain about whether my understanding is correct and it is so limited that reproduction of the algorithm is not possible. Furthermore, the reader should not be required to reconstruct the idea from a paper from the details.\nTherefore, an evaluation of the contribution and significance is not really possible for me. As I cannot **even** get a rough understanding from reading the paper and googling related work / textbooks, I think the paper needs to be rewritten to make it suitable for the ICLR community. If I understood the approach partially correctly, I think that the idea is really neat, interesting and novel and gave me a new perspective. However, the experiments also focus on well posed standard problems and not on the more complex problems of multi-body contact right mechanics which is targeted with many of the referred works.\n\nIn the following, I will state my main questions regarding the understanding.\n\n\nSummary & Questions for each section:\n\n(1) Introduction:\nFor me the introduction is missing to state the clear problem statement and the available information to solve the problem. What kind of application do you have in mind? The paper cites a lot of graph networks from Battaglia et. al. or Kipf et. al. but I cannot relate your problem to the works of these authors. I guess you also want to learn a simulator but then the experiments just focus on solving very small PDEs. Besides the main motivation I am missing an exact mathematical problem statement. My guess is that you want to solve for x(p) given the differential equation described by \\partial x / \\partial p = f(p, x(p)). Now the question for me is, what is known / unknown about the differential equation? E.g.,\n* Do I assume to know the structure of f but not the exact parameters?\n* Do I know f but not the solution x(p)?\n* Do I know x(p) at some points and nothing about f?\n\nMy guess is the last, but I am uncertain. Following a precise problem statement, I would like to see the general approach how to solve the problem and how this is currently done. For this paper my guess is that the standard approach to obtain the PDE solutions is to derive the filter C manually from the analytical f. Then one can solve for the solution of x(p), by convolving C with the mesh of x_k obtaining A and obtaining new x_{k+1} by solving A x_{k+1} = b. Is that correct? Furthermore, I am uncertain about the definitions of b and x. I guess x is a vectorized mesh of the domain of x. Such an short theoretical overview of standard approaches would be very helpful as the google results of picard iteration lead to very different symbolic approaches.\n\n(2) Method:\nI would like to understand the general algorithm first and then deep dive into the details. The current version just presents the details but not the overall approach. Hence the reader has to reconstruct the main approach from the details, which is really cumbersome. My current understanding of your approach is that you start with a random neural network for C and perform the picard iteration with the random C. At the end of the picard iteration you compute the MSE of the predicted solution and the observed solution and update the filter C to minimize the mse at the end of the picard iteration. To compute the derivatives about the iterative picard solver you use the adjoint method. I am not certain if I understood it correctly or not.\n\n(3) Network Architecture:\nThe paper uses a uncommon optimization scheme for deep networks, i.e., IPopt + Adam. And the experiments show that pure Adam does not work, could the authors please discuss the differences between IpOPT and Adam and discuss why pure Adam does not work?\n\n(4) Experiments:\nThe section provides an overview about simple 1d and 2d PDE problems. The data samples are really small 4 - 6 samples (if I understand correctly) and the obtained performance is outrageously good mse, i.e., 10^-14. This seems a bit fishy to me and proposes that the experiments are way too simple and the provided data is way too perfect. The problem with real-world data is that it is really noisy and one would need to recover the structure despite the noise. Therefore, I would like to see this approach applied to much more challenging problems, preferably with real data.\n\n(5) Related Work:\nAs mentioned before the connections to other works are not really made clear, especially to the application centric graph networks.  \n\nOne more general question:\nIf I understand the motivation correctly, is this approach also applicable to controlled systems as the controller would change the dynamics frequently and then the filter C would need to be relearned in each iteration of the policy optimization?\n\n\n**Post Rebuttal Comments**: The authors improved the paper during the rebuttal but the clarity is not sufficient and the results are still puzzling. I do not fully understand how one can learn the perfect solution with only 3-4 data points.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8YhAYkXu_s2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-discussion update: The authors only partially adressed my concerns in their rebuttal. The paper suffers from lack of comparisons: only 2 baselines are compared, and only on few systems. Crucially the new Navier-Stokes experiment lacks comparisons. The authors also couldn't respond to my questions about research context or scope: it's difficult to assess what this work actually claims in relation to competing methods. For a machine learning paper this is not enough.\n\n-----\n\nThe paper proposes input-dependent convolutions for PDE learning under Picard solvers. The main contribution seems to be the spatially non-stationary filters, which is a useful and somewhat known CNN technique. It is easy to see how this makes the neural PDEs more powerful, but it’s unclear in which setting the spatially-evolving differentials are warranted in PDEs. The paper should discuss the motivation and justification of this choice more. \n\nIt’s unclear if the adjoints or picard solver are novel or adaptations from earlier works.\n\nI wonder why the spatial convolutions do not seem to overfit. They can learn in principle arbitrarily complex mappings, and with scarce data should overfit badly. How was this handled?\n\nThe paper is written in a clear manner, but almost all math suffers from undefined symbols and variables, which makes the math frustrating to read. Also the problem domain and problem definitions are undefined.\n\nThe experiments show that the method perfectly learns example systems with practically 0 test error. While this sounds great, this also raises a lot of questions. Perhaps the problems were trivial to begin with (there are no comparison to other methods), or maybe they do not generalize (there are no extrapolation experiments to regions where no data has been observed). The problems were also very small (1D or 2D), so are they practically relevant or do they actually require neural machinery of this kind? The author’s should present learning curves and ablation studies to show when how far the method can extend before breaking (fewer data, more noise, more dimensions, more complex systems, etc), and include uncertainty analyses. Extensive comparisons to other methods (both neural and non-neural) are necessary to demonstrate useful contribution. Finally, it’s unclear if these results indicate generalisation (in some sense). \n\nAll experiments seem almost textbook examples of simple, regular and clean cases. Neural networks excel when applied to messy and complex problems. Why are these a good application of neural PDEs?\n\nThe paper presents an incremental convolution extension to learning of PDE convolutions in Picard setting. The results show fantastic performance, which however is undermined by lack of any comparisons and the almost toy-likely simple systems that were studied. \n\n\n\nTechnical comments\no The functional C is undefined and unexplained, what does it mean? It seems to be a “form” instead of differential. How does C relate to PDEs?\no Also “A” and “b\" are explained, how do they relate to PDEs?\no The notation throughout needs to be better defined, eg. domains of all variables\no The paper is lacking the basic PDE definitions completely. Please define the problem this paper is tackling\no Clarify what is “PDE unknown x”. I assume it's the solution.\no What is “current state of x”? Isn’t “x” the state itself?\no eq1 is not a function of “p” but it still uses “p”, please fix\no eq1 seems to redefine C(x,p) to C({x},{p}). How do we handle neighborhoods? How are the neighborhoods defined (sets/subgraphs/tensors?) What does the [..](pj) notation mean? How come we have x^t here, despite A(..) not being a function of time. What is the domain/codomain of C? Please clarify\no Please define x_0,x (is it a matrix?) \no what is “feature map” in eq 2?\no define “w” in eq2\no I wonder if w(x_mn) would have been more intuitive notation for input-dependent w-functions\no N = {} is either a set or flattened vector, not both\no It would be more sensible to define N and C as matrices than vectors since they are tensors with two indices\no what is x_n (is “n” a time or something else?)\no Is the “A” in eq5 the “A” in eq 2 or 3? Where is the “b” coming from, it’s not defined\no Is the adjoint derivation 2.3. novel? \no What is the “MSE” in experiments? Please report training and testing errors separately\no I don’t understand why a time-dependent wave equation was tested, since the model is not time-dependent but spatially dependent. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The objective of the paper is to develop a framework for solving PDES with reduced model size and for scarce observation settings. It proposes to use functional input dependent convolutions for learning spatio-temporal differential operators together with a non linear numerical scheme (Picard solver). Training makes use of an adjoint formulation.

All the reviewers agree that the authors improved the initial version but opt for a reject. In its present form, the technical description is still incomplete with missing explanations. The experiments should be reinforced and the results are partly unexplained.
Decision: Reject