Neural Time-Dependent Partial Differential Equation
Authors: Yihao Hu
Tong Zhao
Zhiliang Xu
Lizhen Lin
Venue:  (2021)
Keywords: Numerical analysis
Deep learning
Partial differential equation
Machine learning
Predictive modeling

Abstract: Partial differential equations (PDEs) play a crucial role in studying a vast number of problems in science and engineering. Numerically solving nonlinear and/or high-dimensional PDEs is frequently a challenging task. Inspired by the traditional finite difference and finite elements methods and emerging advancements in machine learning, we propose a  sequence-to-sequence learning (Seq2Seq) framework called Neural-PDE, which allows one to automatically learn governing rules of any time-dependent PDE system from existing data by using a bidirectional LSTM encoder, and predict the solutions in next $n$ time steps. One critical feature of our proposed framework is that the Neural-PDE is able to  simultaneously learn and simulate all variables of interest in a PDE system. We test the Neural-PDE by a range of  examples, from one-dimensional PDEs to a multi-dimensional and nonlinear complex fluids model. The results show that the Neural-PDE is capable of learning the initial conditions, boundary conditions and differential operators defining the initial-boundary-value problem of a PDE system without the knowledge of the specific form of the PDE system. In our experiments, the Neural-PDE can efficiently extract the dynamics within 20 epochs training and produce accurate predictions. Furthermore, unlike the traditional machine learning approaches for learning PDEs, such as CNN and MLP, which require great quantity of parameters for model precision, the Neural-PDE shares parameters among all time steps, and thus considerably reduces computational complexity and leads to a fast learning algorithm. 

Under review as a conference paper at ICLR 2021
NEURAL TIME-DEPENDENT PARTIAL DIFFERENTIAL EQUATION
Anonymous authors Paper under double-blind review
ABSTRACT
Partial differential equations (PDEs) play a crucial role in studying a vast number of problems in science and engineering. Numerically solving nonlinear and/or high- dimensional PDEs is frequently challenging task. Inspired by the traditional finite difference and finite elements methods and emerging advancements machine learning. we propose sequence-to-sequene Neural-PDE, which allows one to automatically learn governing rules of any time- dependent PDE system from existing data by using bidirectional LSTM encoder, and predict the solutions in next n time steps. One critical feature of our proposed framework is that the Neural-PDE is able simultaneously learn and simulate all variables of interest in a PDE system. We test the Neural-PDE by a range of examples, from one-dimensional PDEs multi-dimensional and nonlinear complex fluids model. The results show that the Neural-PDE is capable of learning the initial conditions, boundary conditions and differential operators defining the initial- bitiial-bondrr--alllle problem of PDE system without the knowledge of the specific form of the PDE system. In our experiments, the Neural-PDEE can efficiently extract the dynamics within 20 epochs training and produce accurate predictions. Furthermore, unlike the traditional machine learning approaches for learning PDEs, such as CNN and MLP, which require great quantity of parameters for model precision, the Neural -DE shares parameters among all time steps, and thus considerably reduces computational complexity and leads to a fast learning algorithm.
learning (Seq2Seq) framework called
INTRODUCTION
The research of dependent partial differential equations (PDEs) is regarded as one of the most important disciplines in applied mathematics. PDEs appear ubiquitously in broad spectrum of fields including physics, biology, chemistry, and finance, to name a few. Despite their fundamental importance, most PDEs not be solved analytically and have to rely numerical solving methods. Developing efficient and accurate numerical schemes for solving PDEs, therefore, has been an active research area the past few decades (Courant et al., 1967: Osher & Sethian, 1988; Le LeVeque Cockburn et al., 2012; Thomas, 2013; Johnson, 2012). Still, devising stable and accurate schemes with acceptable computational cost a difficult task, especially when nonlinear and(or) high-dimensional PDEs are considered. Additionally, PDE models emerged from science and engineering disciplines usually require huge empirical data for model calibration and validation, and determining the multi- dimensional parameters in such PDE system poses another challenge (Peng et al., 2020).
Deep learning is considered be the state-of -ft----rtttttttt- tool in classification and prediction of nonlinear inputs, such as image, text, and speech (Litjens et al., 2017: Devlin et al.. 2018; LeCun et al., 1998; Krizhevsky et al., 2012; Hinton al., 2012). Recently, considerable efforts have been made to employ deep learning tools in designing data-driven methods for solving PDEs (Han et al., 2018; Long et al., 2018; Sirignano Spiliopoulos, 2018; Raissi et al., 2019). Most of these approaches are based on fully-connected neural networks (FCNNs), convolutional neural networks (CNNs) and multilayer perceptron (MLP). These neural network structures usually require an increment of the layers to improve the predictive accuracy (Raissi et al., 2019), and subsequently lead to more complicated model due to the additional parameters. Recurrent neural networks (RNNs) are one type of neural network architectures. RNNs predict the next time step value by using the input data from the current

---

Under review as a conference paper at ICLR 2021
and previous states and share parameters across all inputs. This idea (Sherstinsky. 2020) of using current and previous step states to calculate the state at the next time step is not unique to RNNs. In fact, it is ubiquitously used in numerical PDEs. Almost all time-stepping numerical methods applied to solve time-dependent PDEs, such as Fuler's, Crank-Nicolson, high-order Taylor and its variance Runge- Mutta-Kutaa (Ascher et al., 1997) time-stepping methods, update numerical solution by utilizing solution from previous steps.
This motivates us to think what would happen if we replace the previous step data in the neural network with numerical solution data to PDE supported on grids. It is possible that the neural network behaves like time-stepping method, for example. forward Euler's method yields the numerical solution at a new time point as the current state output (Chen et al., 2018). Since the numerical solution on each of the grid point (for finite difference) or grid cell (for finite element) computed at set of contiguous time points can be treated neural network input in the form of one time sequence of data, the deep learning framework can be trained to predict any time-dependent PDEs from the time series data supported on some grids if the bidirectional structure is applied (Huang et al., 2015; Schuster & Paliwal, 1997). In other words, the supervised training process can be regarded as practice of the deep learning framework to learn the numerical solution from the input data, by learning the coefficients on neural network layers.
Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) is neural network built upon RNNs. Unlike vanilla RNNs, which suffer from losing long term information and high probability of gradient vanishing or exploding, LSTM has specifically designed memory cell with set of new gates such as input gate and forget gate. Equipped with these new gates which control the time to preserve and pass the information, LSTM capable of learning long term dependencies without the danger of having gradient vanishing or exploding. In the past two decades, LSTM has been widely used in the field of natural language processing (NLP), such as machine translation, dialogue systems, question answering systems (Lipton et al., 2015).
Inspired by numerical PDE schemes and LSTM neural network, we propose a new deep learning framework, denoted as Neural-PDE. It simulates multi-dimensional governing laws, represented by time-dependent PDEs, from time series data generated on some grids and predicts the next n time steps data. The Neural-PDE is capable of intelligently processing related data from all spatial grids by using the bidirectional (Schuster & Paliwal, 1997) neural network, and thus guarantees the accuracy of the numerical solution and the feasibility in learning any time-dependent PDEs. The detailed structures of the Neural-PDE and data normalization are introduced in Section 3.
The rest of the paper is organized as follows. Section 2 briefly reviews finite difference method for solving PDEs. Section 3 contains detailed description of designing the Neural-PDE, In Section and Appendix A of the paper, we apply the Neural-PDE to solve four different PDEs, including the I-dimensional(1D) wave equation, the 2-dimensional(2D) heat equation, and two systems of PDEs: the invicid Burgers' equations and a coupled Navier Stokes-Cahn Hilliard equations, which widely appear in multiscale modeling of complex fluid systems. We demonstrate the robustness of the Neural-PDE, which achieves convergence within 20 epochs with an admissible mean squared error, even when we add Gaussian noise in the input data.
2 PRELIMINARIES
2.1 TIME DEPENDENT PARTIAL DIFFERENTIAL EQUATIONS
A time-dependent partial differential equation is an equation of the form:
du dz Dan
Ou
82u
82u
2m
U₁ = f ... U,
(2.1.1)
Dz dzydni
Day Or.
Orn
where u = wtt,, 21, nn)) is known, xy E R are spatial variables, and the operator f maps R + R. For example, consider the parabolic heat equation: in a²Au, where u represents the temperature and ff is the Laplacian operator A. Eq. (2.1.1) can be solved by finite difference methods, which is briefly reviewed below for the self-commpeteness of the paper.
2

---

Under review as a conference paper at ICLR 2021
2.2 FINITE DIFFERENCE METHOD
Consider using finite difference method (FDM) to solve a two-dimensional second-order PDE of the form:
Ut = (x, M, My Mawy Wyy); (xr, v) E EQCR? (0))
(2.2.1)
with some proper boundary conditions. Let Q be Q [Ta: zbl [Ma. yb). and
" = urii, M.).
(2.2.2)
where i, = not, 0 S n S N, and öt = T for E [0, T], and some large integer N. 2; = iδx, 0 <
i I No. dx Fam for x E [b). Vj joy, 0 S Ny dy y. N, for y E Wa: Ub) N and Ny are integers.
The central difference method approximates the spatial derivatives as follows (Thomas, 2013):
1
(Ti, Uj.
(Ui+1. (4-1,5) + (bx²")
(2.2.3)
20x
1
(vi, Uj. 1)
(wi5+] - -wij-1) + (6)²²)
(2.2.4)
2ōy
1
Uxx (Zt, yj. t)
(14+1, - -2u, + 14-1,1) ) O(8x²)
(2.2.5)
8x2
My (zi, Uj. t) =
(Wi.j+1
2us, + Mi.j + (6))))
(2.2.6)
8y²
To this end, the explicit time-stepping scheme to update next step solution *1+
+1 is given by:
Uii ~ U UiJ + 8tf( Uj, U U -i Ui Utt1j 1T
(2.2.7)
F F( Uj. 5x, dy, St, U U - U ++ 1 1
(2.2.8)
Apparently, the finite difference method (2.2.7) for up- dating w=11 on grid point relies on the previous time steps' solutions, supported on the grid point and its neigh- bours. The scheme (2.2.7) updates w n++1 using four points of un values (see Figure 1). Similarly, the finite element method (FEM) approximates the new solution by calculat- ing the corresponded mesh cell coefficient (see Appendix), the
j+1
111
which is updated by its related nearby coefficients mesh. From this perspective, one may regard the numeri- cal schemes for solving time-dependent PDEs as methods catching the information from neighbourhood data of in- terest.
01
Figure 1: updating scheme for central difference method
PROPOSED METHOD
3.1 MATHEMATICAL MOTIVATION
Recurrent neural network including LSTM an artificial neural network structure of the form (Lipton al., 2015):
h' = @Whh=g' whhp + b.) = (x .hh hh'1'11== a3(a² ,,,
x²)
(3.1.1)
where at Rd is the input data of the 8th state and h R denotes the processed value in its previous state by the hidden layers. The output y of the current state is updated by the current state value h':
y' = =oWhhn b,) = =oc(h') = a.( x x² "..,,,")
(3.1.2)
(3.1.3)
Here Who E Raxd Wh E g why E k are the matrix of weights, vectors b, b, E R the coefficients of bias, and a, Ja, Tb, e; aa corresponded activation and mapping functions.
3

---

Under review as a conference paper at ICLR 2021
With proper design of imput and forget gate, LSTM effectively yield better control over the gradient flow and better preserve useful information from long-range dependencies (Graves & Schmidhuber, 2005).
Now consider temporally continuous vector function u E _ given by an ordinary differential equation with the form:
du((t) g((()))
(3.1.4)
di
Let un ntt nôt), forward Euler's method for solving u can be easily derived from the Taylor's theorem which gives the following first-order accurate approximation of the time derivative:
du n+ ua + ((0t)
(3.1.5)
di
81
Then we have: du
g(u) (3.1.5)
u n+1 = un + +o g("") 0(8t))
dt
+
f1 oi o... fi
(3.1.6)
Here û" u((ôtt the numerical approximation and fi = u² ot g( RR R" Combining equations (3.1.1) and (3.1.6) one may notice that the residual networks, recurrent neural network and also LSTM networks can be regarded equations if more layers are added and smaller time steps are taken. (Chen et al., 2018)
numerical scheme for solving time- -eneeednddnt differential
Canonical structure for such recurrent neural network usually calculate the current state value by its previous time step value h and current state input æ'. Similarly, in numerical PDEs, the next step data at grid point is updated from the previous (and current) values on its nearby grid points (see Eq. 2.2.7).
Thus, what we replace the temporal input h and xt with spatial information? A simple sketch of the upwinding method for a 1d example of u(x, l):
Ut + vug =
(3.1.7)
will be:
u1+1 = ut dx (u," - u.-1) + (8xx 5t) - 0,5 f2(06111 0.
(3.1.8)
fo (f (z,, h h fo.n ( (8) 0p -/-1: 0.) 2/ 1/-11
(3.1.9)
8. 0,' h (ū) o (ū 2 (ū 0f 02
(3.1.10)
Here let ++1 be the prediction of ag processed by neural network. We replace the temporal previous state with spacial grid value h;-1 and input the numerical solution of s u(iōæ, nol) as current state value, which indicates the neural network could be seen as a forward Euler method for equation 3.1.7 (Lu et al., 2018). Function fa ú." - v 5t (α. 1) : IR + R and the function fo represents the dynamics of the hidden layers decoder with parameters 0, and fn specifies the dynamics of the LSTM layer (Hochreiter & Schmidhuber, 1997; Graves & Schmidhuber, 2005) in encoder withe parameters 17. The function fa., simulates the dynamics of the Neural-PDD with paramaters 0 and 17. By applying Bidirectional neural network, all grid data are transferred and it enables LSTM to simulate the PDEs as:
0 (f ūT, (())(a))
(3.1.11)
het1(u)= fo (( 2/42 0/ (2
(3.1.12)
For a time-dependent PDE, if we map all our grid data into an input matrix which contains the information of ōz, öt, then the neural network would regress such coefficients as constants and will learn and filter the physical rules from all the is mesh grids data as:
of a+1 fe.n (i: 0f. uz.... Ag
(3.1.13)
The LSTM neural network is designed to overcome the vanishing gradient issue through hidden layers, therefore such recurrent structure to increase the stability of the numerical approach in deep learning. The highly nonlinear function fo.,, simulates the dynamics of updating rules for t which works in way similar finite difference method (section 2.2) finite element method.
4

---

Under review as a conference paper at ICLR 2021
BLSTM Decader
LSTM
1STM
ISTT
ISTM
LSTM
ISTM
BESTM Encader
Figure 3: Neural-PDE
3.2 NEURAL-PDE
In particular, we use the bidirectional LSTM (Hochreiter Schmidhuber, 1997; Graves & Schmidhuber, 2005) to better retain the state information from data on grid points which are neighbourhoods in the mesh but far away in input matrix.
time
The right frame of Figure shows the overall de- sign of the Neural-PDE. Denote the time series data at collocation points as aÑ, a ,..., with af [ūg, út, ... " at 7th point. The superscript represents different time points. The Neural-PDE takes the past states {aī" a. ..... a } of all collocation points, and out- puts the predicted future states (bM bb .... b M1. where b,M = [w] N+1 2/ NN ..... n, M is the Neural-PDE predic- tion for the 7th collocation point at time points from N + 1 to N M. The data from time point 0 to N are the training data set.
dataa
[2g 2f, 22 ..., [2
The Neural-PDE is incoder-decode style sequence model that first maps the input data to a low dimensional latent space that
example of maping 2d data matrix into 1d vector where k = N₂ x N, and N, and N,, are the num- bers of grid points on x and respec- tively.
Figure
h; = I.ST)((.)) @ LSTM((;))
(3.2.1)
where @ denotes concatenation and h; is the latent embed- ding of point a; under the environment.
One then decoder, another bi-lstm with dense layer:
0 [..TM((h:) @ fsTM(b))) w,
(3.2.2)
where Wi is the learnable weight matrix in the dense layer.
During training process, mean squared error (MSE) loss L is used as we typically don't 't know the specific form of the PDE.
N+M k
C
1/
(3.2.3)
N=
1=1
3.3 DATA INITIALIZATION AND GRID POINT RESHAPE
In order to feed data into our sequence model framework, we map the PDE solution data onto a K N matrix, where K E Z is the dimension of the grid points and N E Z+ is the length of the time series data on each grid point. There no regularization for the input order of the grid points data in the matrix because of the bi-directional structure of the Neural-PDE.. For example, 2d
5

---

Under review as a conference paper at ICLR 2021
1d Wave
2d Heat
2d Burgers"
Fluid System
MSE
7.4444A
5 7.0741E -
1.4018E
6.163IE -
Table Neural-PDr shows very small test MSE on 4 different PDEs.
Allen-Cate Cahn
Burgers
PINN
7.0E
6.7E 4
Neura-PDE
2.9E
2.4E
Table 2: Neural-PDE outperforms baseline in test MSE on 1d Allen-Cahn and Burgers equations.
heat equation at some time t is reshaped into ld vector (See Fig. 2). Then the matrix is formed accordingly.
For re-dimensional time-dependent partial differential equation with K collocation points, the input and output data for t E (0, I)) will be the form:
a
A(K,N)
6
07
6/
(3.3.1)
K
K
6/
N+
+m
N+
B)
(3.3.2)
N+-M
Here N I and each row é represents the time series data the 8th mesh grid, and M is the time length of the predicted data.
By adding Bidirectional LSTM encoder in the Neural-PDE, it will automatically extract the informa- tion from the time series data as:
B(K, M) EDESolver(A(K,, = PDESolver(a) at ax" ,ax) (3.3.)
COMPUTER EXPERIMENTS
sequence to sequence learning framework which allows one to predict within any time period by the given data. One may test the Neural-PDE using different permutations of training and predicting time periods for its efficiency, robustness and accuracy. In the following examples, the whole dataset is randomly splitted in 80% for traning and 20% for testing. We will predict the next tp E [31 x öt, 40 x čt) PDE solution by using its previous f,r E [0, 30 x ot] data as:
Since the Neural-PP
B(K, 10) PDESoluer(A(K, (A(,,
(4.0.1)
Table 1 summaries the experimental results of the Neural-PDE model on 4 different PDEs, which achieve extremely small MSEs from 10-5 ~ 10-7 Table shows the comparison results of our proposed Neural-PDE with the state-of-the-art method Physically Informed Artificial Neural Networks (PINN) (Raissi et al., 2019) on two PDEs (1d Allen-Cahn and 1d Burgers' equation). Neural-PDE is able to outperform PINN while having much less parameters, where PINN contains 4 hidden layers with 200 neurons per layer and Neural-PD -DE only consists of 3 layers (2 bi-lstm with 20 neurons per layer and 1 dense output layer with 10 neurons).

---

Under review as a conference paper at ICLR 2021
octine
Deta
(a) Exact utt 1)
(b) Predicted u(t = 1)
(c) Exact v(t = 1)
(d) Predicted u(t 1)
Figure 4: Neural-PDE shows ideal prediction on Burgers' equation.
EXAMPLE: INVISCID BURGERS' EQUATION
Inviscid Burgers' equation is a classical nonlinear PDE in fluid dynamics. In this example, we consider an invicid Burgers' equation which has the following hyperbolic form:
du du
du
de
0t + u du
du
+ u Ox
+ U
= O,
+ U = 0
(4.0.2)
0t
dy
8y
Ω= [0 [0. 1), [0. 1],
(4.0.3)
and with initial and boundary conditions:
(0.25 < x < 0.75, 0.25 < y S 0.75, ( 0) 0.9
(4.0.4)
8 (0.25 < S 0.75, 0.25 1 y S 0.75, 0) 0.5
(4.0.5)
((. y. t) = u(1,
u(z, u(z, 1,t) =
(4.0.6)
The invicid Burgers' equation is hard to deal with in numerical PDEs due the discontinuities (shock waves) in the solutions. We use upwinding finite difference scheme to create the training data and put the velocity u, v in to the input matrix. Let 8x = 6y 10 2 öt 10-3, our empirical results (see Figure 4) show that the Neural-PDE is able to learn the shock waves, boundary conditions and the rules of the equation, and predict u and simultaneously with an overall MSE of 1.4018 10-5. The heat maps of exact solution and predicted solution are shown in Figure 5.
EXAMPLE: MULTISCALE MODELING: COUPLED CAHN-HILLIARD HILLLLLLLDDAAIIIEE -TOKESS SYSTEM
Finally, let's consider the following 2d Cahn--iilllrddNaavie---tto system widely used for model- ing complex fluids:
Ut + u Vu = - Vp + vsu VVH,
(4.0.7)
Φ: + (uØ) M An,
(4.0.8)
/ X( AΦ
n (#2-1)) 1))
(4.0.9)
u
(4.0.10)
In this complicated example we will use the following initial condition
1 (1, v, 0) = - 50 tanh(Ji - 0.1)) + 1 - 50 tanh( - 0.1)), T.C.
(4.0.11)
fi V 0012) + f2 v(z 0.12)² (1)² with 0.5, 1-0.5 0.5]. E [o, 11, M = 0.1, v 0.01
(4.0.12)
(4.0.13)
This fluid system can be derived by the energetic variational approach (Forster, 2013). The complex fluids system has the following features: the micro-structures such as the molecular configurations, the interaction between different scales and the competition between multi-phase fluids (Hyon et al., 2010). Here u is the velocity and 6(I, y, t) E [0, 1] denotes the volume fraction of one fluid phase. M is the diffusion coefficient and μ is the chemical potential of o. Equation (4.0. 0010) indicates the incompressibility of the fluid. Solving such PDE system is notorious because of its high nonlinearity and multi physical and coupled features. One may use the decoupled projection method (Guermond al., 2006) to numerically solve it efficiently or implicit method which however is computationally
7

---

Under review as a conference paper at ICLR 2021
Capc Dara
Nradcae Data
(a) Exact Test Dataset (b) Predicted Test Dataset
(a) Exact Test Dataset
(b) Predicted Test Dataset
Figure 5: Neural-PDE shows ideal prediction on 2d Burgers Equation.
Figure Neural-PDE shows ideal prediction on Fluid System.
00
8
8
Figure 7: Predicted data by Neural-PDE (first row) and the exact data (second row) of volume fraction (column 1-3) and pressure p (column 4-6). The graphs of each columns 1-3 and 4-6 represent the time states of é1, 12, t3, where 0 S t < t2 < t3 S 1.
expensive. Another challenge of deep learning in solving a system like this is how to process the data to improve the learning efficiency when the input matrix consists of variables such large magnitude value and variable of very small values such as p ~ 10 For the Neural-PDE to better extract and learn the physical features of variables in different scales, normalized the data with a sigmoid function. Set öt x 10-4 and here the training dataset is generated by a FEM solver FreeFem++ (Hecht, 2012) using Crank-Nicolson finite element scheme. Our Neural-PDE prediction shows that the physical features of p and ø have been successfully captured with an overall MSE: 6.1631 10-7 (see Figure 7).
[0, 1] with
CONCLUSIONS
In this paper, we proposed novel sequence recurrent deep learning framework: Neural-PDE, which is capable of intelligently filtering and learning solutions of time-dependent PDEs. One key innovation of our method is that the time marching method from the numerical PDEs is applied in the deep learning framework, and the neural network is trained explore the accurate numerical solutions for prediction.
The satte-ofthh-azz researches have shown the promising power of deep learning in solving high- dimensional nonlinear problems in engineering, biology and finance with efficiency in computation and accuracy prediction. However, there are still unresolved issues in applying deep learning in PDEs. For instance, the stability and convergence of the numerical algorithms have been rigorously studied by applied mathematicians. Due to the high nonlinearity of the neural network system and the curse of dimensionality (Hutzenthaler et al., 2019), theorems guiding stability and convergence of solutions predicted by the neural network are to be revealed.
Lastly, it would be helpful and interesting if one theoretically characterize numerical scheme from the neural network coefficients and learn the forms or mechanics from the scheme and prediction. We leave these questions for further study.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Uri M Ascher, Steven Ruuth, and Raymond J Spiteri. Implicit-expiii runge-kutta methods for time-dependent partial differential equations. Applied Numerical Mathematics, 25(2-3):111-167, 1997.
Fischer Black and Myron Scholes. The pricing of options and corporate liabilities. Journal of political economy, 81 (3))6337-654 1973.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571 3571-6583, 2018.
Bernardo Cockburn, George E Karniadakis, and Chi-Wang Shu. Discontinuous Galerkin methods: theory, computation and applications, volume 11. Springer Science & Business Media, 2012.
Richard Courant, Kurt Friedrichs, and Hans Lewy. On the partial difference equations of mathematical physics. IBM journal of Research and Development, 11(2):215-234, 1967.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 111.:00810004805 2018.
Johannes Forster. Mathematical modeling of complex fluids. Master's, University of Wurzburg. 2013.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6):602-610, 2005.
Jean-Luc Guermond, Peter Minev, and Jie Shen. An overview of projection methods for incompress- ible flows. Computer methods in applied mechanics and engineering. 195(44-47):601 1-6055, 2006.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 11:85505 8510, 2018.
Hecht. New development freefem++. Numer. Math., 20(3-4):251-265. 2012. ISSN 1570-2820.
URL https:tpph... / free fem. org/.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):2-997, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1780- 1997.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging. arXiv preprint (XX::15008.099/ 2015.
Martin Hutzenthaler, Arnulf Jentzen, Thomas Kruse, and Tuan Anh Nguyen. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations, 2019.
Yunkyong Hyon, Chun Liu, et al. Energetic variational approach in complex fluids: maximum dissipation principle. Discrete Continuous Dynamical Systems-A, 26(4):1291, 2010.
Claes Johnson. Numerical solution of partial differential equations by the finite element method. Courier Corporation, 2012.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1 1005 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11) (6(11):277823244 1998.

---

Under review as a conference paper at ICLR 2021
Randall J Vequee. Numerical methods for conservation laws, volume 3. Springer.
Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv: 1 1506000199 2015.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60-85 88, 2017.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216, 2018.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285, 2018.
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5528-5531. IEEE, 2011.
Stanley Osher and James A Sethian. Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi formulations. Journal of computational physics, 79(1):12-49 1988.
Grace CY Peng, Mark Alber, Adrian Buganza Tepole, William R Cannon, Suvranu De, Savador Dura- Bura-Bemnal, Krishna Garikipati, George Karniadakis, William W Lytton, Paris Perdikaris, et al. Multiscale modeling meets machine learning: What
lear?? Archives of Computational
Methods in Engineering, pp. 1-21, 2020.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks; A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions Signal Processing, 45(11))2673-2665 1997.
Alex Sherstinsky. Fundamentals recurrent neural network (rnn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364 2018.
James William Thomas. Numerical partial differential equations: finite difference methods, vol- ume 22. Springer Science & Business Media, 2013.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1
FINITE ELEMENT METHOD
Finite element method (FEM) is powerful numerical method in solving PDEs. Consider ID wave equation of uxx t):
uu 0 Ugg f. x zE [a, = 22 te
(A.1.1)
Us (a t) uz =0
(A.1.2)
The function is approximated by function un
N
m( (n Σ (.(()(())))
(A.1.3)
(A.1.4)
where v/, V, is the basis functions of some FEM space V, and af denotes the coefficients. N denotes the degrees of freedom. Multiply the equation with an arbitrary test function dh and integral over the whole domain have:
[ ueet de + u² / VuVuly dr = 1 f4g dr
(A.1.5)
(A.1.6)
and approximate u(x, 1) by up:
N
aa,(t) (t)
( weo dx +u? L (t) / Vei. de
/ 1
Σ
fVs dr,
(A.1.7)
dt
M
Aij
M at + V'A
(A.1.8)
Here M is the mass matrix and A is the stiffness matrix, a = (a1..., o x is a N x 1 vector of the coefficients time The central difference method for time discretization indicates that (Johnson, 2012):
a+ =2a" ---1__ M 1 (b aA
(A.1.9)
n+1
w/
a+1
a (1(x))))
(A.1.10)
A.2 LONG SHORT-TERM MEMORY
Long Short-Tern Term Memory Networks (LSTM) (Hochreiter & Schmidhuber, 1997; Graves & Schmid- huber, 2005) are a class artificial recurrent neural network (RNN) architecture that is commonly used for processing sequence data and can overcome the gradient vanishing issue in RNN. Similar to most RNNs (Mikolov et al., 2011 1), LSTM takes sequence {x1, Z2, xx) as input and learns hidden vectors (h1, h2 ... h1 for each corresponding input. In order to better retain long distance information, LSTM cells are specifically designed to update the hidden vectors. The computation process of the forward pass for each L.STM cell is defined as follows:
é, = a (W("),
sr + w!!)
h + w(() c-1 + b.),
f: a (W))) (0) xt W(h) h4-1 w a-1 b,)
C: = f:c.-1 + i tanh
(( x + (
h + ba)
0: a (W mx w()) h, w c ba);
h, = o. tanh(c.))
where is the logistic sigmoid function, Ws weight matrices, bs are bias vectors, and subscripts i, f. o and e denote the input gate, forget gate, output gate and cell vectors respectively, all which have the same size hidden vector h.
This LSTM structure is used in the paper to simulate the numerical solutions of partial differential equations.
11

---

Under review as a conference paper at ICLR 2021
A.3 EXAMPLES
A.3.1 WAVE EQUATION
Consider the 1d wave equation:
Met = cu I E [0, 11, t t [0,2]
(A.3.1)
u(r, 0) (An()
(A.3.2)
u(0,t) = u(1,t)
(A.3.3)
Let 1 and use the analytical solution given by the characteristics for the training and testing data:
1 (sim(4Ax + () + sin(4xx t))
m( 1) = 2
(A.3.4)
MOat DAAA
Prodited Data
3.1
124
(a) Exact Test Dataset
(b) Predicted Test Dataset
(c) Training Mctries
Figure 8: dr x 10 öt = 1 x 10-3, MSE: 7.4444 x 10-5, the whole time period length is n4 = 2000 and the mesh grid size 101, the test dataset size 14 and thus the total discrete testing time period is 140, figure (a) and figure(b) are the heat map for the exact test data and predicted test data. Fig shows both training and cross-validation errors of Neural-PDE convergent within 10 epochs.
(a) 1.991
(b) t 1.994
(c) = 1.997
(d) t
Figure 9: We selected the final states for computation and the results indicate that Neural-PDE is robust in capturing the physical laws of wave equation and predicting the sequence time period.
A.3.2 HEAT EQUATION
The heat equation deseribes how the motion diffusion of a heat flow evolves over time. The Black-Scholes model (Black & Scholes, 1973) is also developed based on the physical laws behind the heat equation. Rather than the ID case that maps the data into a matrix (??) with its original spatial locations, the high dimensional PDEs grids are mapped into matrix without regularization of the position, and the experimental results show that Neural-PDE is able to capture the valuable features regardless of the order of the mesh grids in the matrix. Let's start with 2D heat equation as follows:
Ue Uzz + Kyys
(A.3.5)
0.9, if (x - 1)2 (y 1)² < 0.25
u(x, 0)
(A.3.6)
0.1, otherwise
Ω [0,2] x [0, e [0,0.15]
(A.3.7)
12

---

Under review as a conference paper at ICLR 2021
Model Lown
0222
0000
11300.0000
A-000 D005
(a) Exact Test Dataset
(b) Predicted Test Dataset
(c) Training Metrics
Figure 10: dz = 0.02, by = 0.02, 5t 10-4, MSE: 7.0741 x 10-4 the size of the test data is 10 and the test time period 140.
Predittdd
1.59
159
350
1.2
125
325-
11.0
3300
0.18
038
64
0.5
166
1.5
2.0
330
80
6.5
1.4
2.6
038
3.
45
3.5
2.4
(a)
(b)
(c)
Figure 11: figure (a) is the exact solution u(x, y, 0.15) at the final state and figure (b) is the model's prediction. Figure (c) is the error map.
13


Reviews:
[{"review_id": "eu6FwLoJ1S9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a sequence-to-sequence approach for learning the time evolution of PDEs. The method employs a bi-directional LSTM to predict solutions of a PDE-based formulation for a chosen number of time steps. By itself this is an interesting, and important goal, but the method does not seem to contain any novel components apart from demonstrating that LSTMs can be used to learn data from PDEs. The paper only compares to a simple form of PINNs, but not to a variety of other time forecasting algorithms available in the deep learning field (LSTM are just one of many methods used these days, a more state of the art one being e.g. transformers). In addition, the examples only contain single cases with relatively simple model equations.\n\nIn terms of exposition, the paper spends a significant amount of space discussing relatively basic finite difference, finite element, and LSTM schemes. \n\nMost importantly, from an ICLR paper, I would expect advances in terms of learning methodology: here the authors use a regular bi-directional LSTM, with a fully supervised loss formulation. No changes or modifications are proposed to improve learning the content of PDE or coupling the PDE formulations with the LSTM. Hence, unfortunately, the submission does not really go beyond a demonstration that LSTMs can learn the PDEs under consideration. While this is good to see, I don't think it's a sufficiently large step forward for recommending acceptance as an ICLR poster.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WqoR7svzhhO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes to use LSTM to learn partial differential equations. Experiments include Wave equation, Heat equation, Burgers' equation, and Navier-Stokes equation. They compare with PINN on Allen-Cahn and Burgers equation.\n\nThe paper is nicely written and easy to understand. But I have several concerns:\n\nNovelty:\nUsing the RNN/LSTM type of networks for time series / time-dependent PDEs doesn't seem to be novel. Convolution-LSTM has been widely used in these tasks e.g. (https://arxiv.org/abs/2002.03014). \nInstead of using Convolution, the author uses the seq-to-seq structure. Unfortunately, there is no sufficient justification or any empirical comparison. Indeed, the seq-to-seq loses the location information compared to convolution-LSTM. I doubt if it can outperform the convolution. \n\nExperiments:\nThe work compares again PINN. In my opinion, it is not a fair comparison. If I understand correctly, the Neural-PDE proposed in this paper uses the ground truth data $u$ for training, but PINN doesn't have the access to these ground truth data. To have a fair comparison, I suggest comparing with LSTM type solvers, for example, Conv-LSTM, Unet-RNN, TF-net (https://arxiv.org/abs/1911.08655)\n\nError Metrics:\nThe paper uses MSE as the loss metric. It's no problem to train with MSE, but when reporting the error rate, it's better to use the relative-L2 error, or at least some relative error. The absolute MSE error looks very encouraging, but it has no physical meaning, in my opinion.\n\nTherefore, I am sorry to suggest rejection.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VYd6D2wXxrb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about using deep neural nets to predict solutions of dynamical systems described by PDEs. The paper is overall well written, and the contribution is clearly stated.\n\nHowever, I do have several issues with the general principle and what is actually achievable with such data-driven approach to solving dynamical systems. Indeed, the idea is as simple as correct; mainly using a DNN with dynamics, e.g., RNNs, to predict solutions of a dynamical systems over time, based on some past (timewise) training data. This is clearly stated in the paper and there is nothing wrong with this idea, except that it seems to me that the authors are exaggerating the value of their findings.\n\nFor instance, when the authors claim that they are learning the law of physics of the PDE, as well as its boundary conditions, they  cannot be further from the truth. Indeed, fitting a predictor based on past data is not a proof of learning anything structural about the PDE, since what you are learning is simply one regime of the solutions, based on the available data. In other words, if you use data points from your transient regime and try to predict the solutions over a time horizon further way from your transient regime, your predictor will fail, simply because you don't have anything in your data to inform the predictor (your DNN) about he future regime ! I hope this makes sense. As a follow up on this point, I want to know in all your examples, what is the time interval over which the learning data is collected ? is it close to the test time instants ? I would expect it to be rather close. Similarly, in your fluid dynamics example, i.e., coupled NS equation, can you please show the results for a more turbulent case, .e.g., $\\mu=1e-3$, for which it is rather impossible to predict the steady state from transient points ! These tests will show you that you are not learning the physics of the system, as you are claiming throughout the paper; please remove these exaggerated statements. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IxGPi0XUzUi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a good contribution to spatio-temporal modeling in complex physics. However, the novelty of the paper is unclear. It is an extended version of Neural-ODEs but used a Bi-LSTM. \n\nPros: A good new approach but very similar to Neural-ODE. There is nothing particularly interesting in this approach.\nCons: Comparison metrics do not show a major difference. $O(10^{-4})$ and $O(10^{-5})$ as RMSE are essentially equal good for a super simple example like inviscid Burgers or any of the other examples. \n\n1) There is a long line of literature that looks at spatio-temporal prediction of chaotic dynamical systems, systems that are more complex than the ones considered in this paper with machine learning and shows state-of-the-art performance with reservoir computers. They do outperform MLP, CNN, and other ML architectures and are much easier to train. **https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.024102; **https://npg.copernicus.org/articles/27/373/2020/; **https://www.sciencedirect.com/science/article/abs/pii/S0893608020300708; **https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2020GL087776. The authors should cite these papers and also consider RC as a baseline. \n\n2) I have some concerns with the comparison with PINNs. There is an advantage to using PINNs that is overlooked by the authors. Since they optimize the residual of the PDEs, they can work without any data at all and can also satisfy BCs by construction. I was wondering how many data points the authors have considered during training PINNs and comparing with Neural-PDEs. \n\n3) Figure 4's quality is very poor. For an average reader, it is impossible for me to make out the axis labels etc. \n\n4) PINNs can be used to satsify BCs by construction. That would undermine one of the points of this paper (satisfying BCs better than PINNs) . I would like to see a plot that shows the how well the BCs are satisfied during inference as we move in time. \n\n5) I would like to see one more test case that is a tad-bit more challenging than the ones considered in this problem. Something chaotic, like Lorenz 96 system can be used. The true potential of the architecture would be revealed when used on a system like that. \n6) Figure 5,6,7 quality is very low. The heat equation considered is too simple for a spatio-temporal prediction problem. There is hardly any major variation with time to pose a challenge. Something simple like the Von-Karman vortex sheet problem in fluid dynamics would be a better test case. The authors should consider looking at test cases (at least a few) to pose a challenge for their architecture\n\n7) I agree with the authors that there is a difference in RMSE when compared to PINNs . PINNs are $\\sim O(10^{-4})$ and their framework is $\\sim O(10^{-5})$. But this just means that both the frameworks are doing amazingly well. The difference would hardly be noticeably, let along have any physical significance for problems such as these. However, this same difference on a very chaotic or turbulent flow would make a difference when it comes to analyzing the long terms statistics. The authors should look at the state-of-the-art of PDE solving with ML and relevant literature that looked at meaningful fluid dynamics/computational physics problems and consider at least one test case that is challenging enough. Example papers: https://arxiv.org/abs/1808.02983, https://arxiv.org/abs/2002.00021", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an approach for learning the dynamics of PDEs. It makes use of bi-directional LSTMs trained to regress future values from past observations, up to a given horizon. Experiments are performed on data generated from numerical solvers on two examples, inviscid Burgers and a Navier-Stokes system. While the topic is fine, the solution is nothing more than regression with sequence models and only shows that RNNs could learn to predict the data generated by these PDEs. The reviewers also highlight that the comparison with the baselines is not appropriate.
Decision: Reject