Discovering Nonlinear PDEs from Scarce Data with Physics-encoded Learning
Authors: Chengping Rao
Pu Ren
Yang Liu
Hao Sun
Venue: ICLR 2022 Poster (2022)
Keywords: Data-driven equation discovery
dynamical system modeling
physics-encoded learning

Abstract: There have been growing interests in leveraging experimental measurements to discover the underlying partial differential equations (PDEs) that govern complex physical phenomena. Although past research attempts have achieved great success in data-driven PDE discovery, the robustness of the existing methods cannot be guaranteed when dealing with low-quality measurement data. To overcome this challenge, we propose a novel physics-encoded discrete learning framework for discovering spatiotemporal PDEs from scarce and noisy data. The general idea is to (1) firstly introduce a novel deep convolutional-recurrent networks, which can encode prior physics knowledge (e.g., known terms, assumed PDE structure, initial/boundary conditions, etc.) while remaining flexible on representation capability, to accurately reconstruct high-fidelity data, and (2) then perform sparse regression with the reconstructed data to identify the analytical form of the governing PDEs. We validate our proposed framework on three high-dimensional PDE systems. The effectiveness and superiority of the proposed method over baselines are demonstrated.

Published as a conference paper at ICLR 2022
DISCOVERING NONLINEAR PDES FROM SCARCE DATA WITH
ENCODED LEARNING
Chengping Rao!, Pu Ren', Yang Liu Northeastern University (rao.che, ren.pu, yang .liu] @@ @northeastern.edu
Hao Sun' Renmin University of China haosun @ @ucc.ddu.en
ABSTRACT
There have been growing interests in leveraging experimental measurements discover the underlying partial differential equations (PDEs) that govern complex physical phenomena. Although past research attempts have achieved great suc- cess in data-driven PDE discovery, the robustness of the existing methods cannot be guaranteed when dealing with low-quality measurement data. To overcome this challenge, we propose novel physics-encoded discrete learning framework for discovering spatiotemporal PDEs from scarce and noisy data. The general idea is to (1) firstly introduce novel deep convolutional- encode prior physics knowledge (e.g., known PDE terms, assumed PDE struc- ture. initial/boundary conditions, etc.) while remaining flexible on representation capability, to accurately reconstruct high-fidelity data, and (2) perform sparse re- gression with the reconstructed data identify the explicit form of the governing PDEs. We validate our method on three nonlinear PDE systems. The effectiveness and superiority of the proposed method over baseline models are demonstrated.
network, which
INTRODUCTION
Deriving physical laws remains critical for understanding the dynamical patterns and complex spa- tiotemporal behaviors in nature. Well formulated governing equations facilitate scientific discovery and promote establishment of new disciplines. Traditionally, scientists seek to find governing equa- tions by rigorously following the first principles, including the conservation laws, geometric brow- nian motion assumptions and knowledge-basee deductions. However, these classical approaches show limited capability and slow progress in identifying clear analytical governing equations, and leave numerous complex systems under-explored. Thanks the recent achievements in machine learning (ML), advances in governing equation/law discovery has been drastically accelerated.
The pioneer works (Bongard & Lipson 2007 Schmidt & Lipson 2009) apply symbolic regres- sion to reveal the underlying differential equations that govern nonlinear dynamical systems with- out any prior background knowledge. Although this inspiring investigation implies the dawn of discovering the fundamental principles automatically in scientific ML, the scalability and overfit- ting issues herein have been the critical concerns. Recently, a ground-breaking work by (Brunton) et 2016) introduces the sparse regression into the discovery of parsimonious governing equations (aka., Sparse Identification of Nonlinear Dynamics (SINDy)), based on the assumption that the dy- namical systems essentially controlled by only few dominant terms, which evolves promote the sparsity in the candidate functions. Furthermore, the introduction of spatial derivative terms and the improvement of the pparsity-promottee (STRidge) make SINDy applicable to general PDE discovery (i.e., the PDE-FIND algorithm) (Rudy al. 2017). However, due to the demanding requirement of derivative estimation via numerical dif- ferentiation, these sparse regression methods largely rely on high-quality measurement data, which barely accessible in the real-world sensing environment. Therefore, there is an urgent need to address the issue of discovering the underlying governing equations from low-resolution (LR) and noisy measurement data. Very recently, many works attempt to relieve the pain by reconstructing the high-resolution (HR) scientific data from L.R measurements and then discovering the governing laws, including robust low-rank discovery (Li al., 20220),, physics-informed spline learning (Sun) et al. 2021) and
algorithm Sequential Threshold Ridge regression
deep learning (Chen al. 2021). Nevertheless, the dominant
"Corresponding author
Equul contribution

---

Published as a conference paper at ICLR 2022
reconstruction part lacks explicit physical interpretability (i.e., grey-box model), which may induce the difficulty in optimization especially for high-dimensional PDE systems with delicate patterns.
Contributions. To overcome the challenges resulting from the noisy and LR measurements, we propose novel physics-encoded deep learning framework (i.e., white-box model) for discovering the governing equations of physical systems. The main contributions of this paper are three-fold:
We design novel recurrent network architecture that is able fit spatiotemporal mea- surement data for HR metadata generation with high accuracy and physical consistency. Notably, this network is characterized with capability encode given physics knowledge (e.g.. known terms, assumed PDE structure based on multiplicative feature operation, ini- tial/boundary conditions) so that the given physics as model prior is respected rigorously.
Based on the proposed reconstruction network, we develop a hybrid optimization approach to discover spatiotemporal PDEs from noisy and LR measurements. Specifically, to deter- mine the explicit form of underlying governing equations, sparse regression is performed by using HR prediction inferred from the model. Finally, through inheriting the PDE terms and coefficients from the sparse discovery, we conduct the fine-tuning step with the com- pletely physics-based network to further improve the accuracy of the scientific discovery.
The proposed coupled framework establishes the physics-encoded data-driven predictive model from the noisy and LR measurement. To evaluate the performance of the model, we perform three numerical experiments on a variety of nonlinear systems. It is found that the proposed approach demonstrates excellent robustness and accuracy in spite of the high level of noise and the scarcity (spatially and temporally) in the measurements.
2 RELATED WORK
Dynamical system modeling. Simulating multi- nulti-dimensional PDE systems with neural networks
(NNs) has been a renewed research topic, which can dated back last century (Lee & Kang. 1990 Lagaris et al. 1998). Like the traditional simulation methods, the recent breakthrough in NN- based modeling for physical systems also fall into two streams: continuous and discrete learning. The continuous learning scheme can avoid the notorious demand of time-step and grid sizes for numerical stability. Among the meshfree strategies, based on the incorporation of prior knowledge of PDE systems, there exist two directions seeking for the approximation of simulation with DNNs, ie., the pure data-driven approaches (Han et al. 2018 Long et al.) 2018: Wang et al.) 2020) and physics-informed learning (Yu et al. 2017 Raissi al. 2019 Rao al. (02 Karniadakis al., 202T]. The former option relies on large amounts of high-quality data, while the physics-informed
ML only requires scarce or even no labeled data due to the enhancement from physical constraints. Besides, it is worthwhile to mention that the recent studies on neural operators (Li et al. 2020b Lu
al. 2021 Patel et al., 2021 have exhibited the great potential of learning the nonlinear, meshfree and infinite-dine dimenseond mapping with DNNs for physical systems.
Regarding the discrete learning scheme, the mesh-based methods play dominant role in simulating complex high-dimensional PDE systems. Herein, the convolutional architectures are mainly applied
for handling regular grids in steady-sh state problems (Zhu & Zabaras 2018 Zhu et al.
2019) and
spatiotemporal systems (Bar-Sinai et al. 2019 Geneva & Zabaras [ Kochkov et al.
2021 Ren
(ct al. 202I), as well as irregular domains by considering elliptic coordinate mapping (Gao et al.
202 Moreover, with the theoretical development of graph neural networks (GNNs) (Kipf & Welling 2016; Bronstein et al. 2017 Battaglia et al., 2018), researchers seek for a new direction for geometry-adaptativ learning of nonlinear PDEs with arbitrary domains (Belbute-Pene
et al.) 2020 Sannhezz-Gnzzlle et al., 2020a) Gao et al.] 2021c). In addition to the common mesh-based approaches, the particle-based frameworks have been attracting lot of interest in scientific model- ing for physical systems, especially by employing GNNs (Li et al. 2018 Ummenhofer et al. 2019 Sanchez-Gonzalez al. 2020b).
Scientific data augmentation. Scientific data are typically sparse, noisy and incomplete. There- fore, data augmentation has been an active research area and received massive attention in the sci- entific ML community, e.g., data reconstruction from sparse measurements (Raissi) 2018 Callaham et al.) 2019 Erichson et al., 2020) |Roo et al. 2021a). super-resolution (SR) (Stengel et al. 2020 Gao et al. 2021b) and denoising (Fathi et al.
2018). Generally, based the type of incorporated
2

---

Published as a conference paper at ICLR 2022
data, we categorize the existing data augmentation methods into two groups. The first one is to extract the coherent structure and correlation features as the basis by utilizing the sufficient high- quality datasets. For instance, in the context of SR. the majority of NN-based frameworks (Liu et al.) 2020 Esmaeilzadeh et al., 2020; Fukami et al. 2021) aim to reconstruct the high-resolution (HR) full-field data from low-resolution (LR) measurements by employing HR ground truth reference as the constraint. Nevertheless, the second type of data augmentation takes advantage of the physics- based model and can accurately capture the high-fidelity dynamical patterns from the LR and noisy measurement data. The recent advances of physics based learning models have shown the the great success of data augmentation for PDE systems with only sparse and noisy data (Raissi 2018 Rao et al. a) and even without any labeled data (i.e., only with PDEs) (Gao et al. 2021b).
Data-driven discovery. The data- discovery also renascent research attempt. The ear- liest work (Dzeroski & Todorovski 1995) introduced the inductive logic programming to find the natural laws from experimental data. With the rapid advance of machine intelligence, the past three decades have seen a considerable amount of literatures growing up around the interplay between ML and scientific discovery. Essentially, two genres have been observed distilling governing equa- tions, with respect to the searching of candidate functions. Firstly, the natural and optimal solution to automatically identify the governing equations for dynamical systems is to learn a symbolic model from experimental data. Hence, symbolic regression (Schmidt & Lipson 2009) and symbolic neural networks (SNN) (Sahoo et al. 2018) Kim et al. (0200) have been explored for inferring the physical models without any prior knowledge. In addition, GNN (Cranmer al., (2200) has also been studied to identify the nontrivial relation due to its excellent inductive capability.
The second group manually defines a large library functions based on the prior knowledge, and uti- lizes the sparse-prompting algorithms to discover the parsimonious physics model efficiently. The most representative works are SINDy (Brunton et al. 2016) and PDE-FIND (Rudy al. 2017) for ordinary differential equations (ODEs) and PDEs, respectively. Nevertheless, the standard sparse representation-baat methods are limited to the high-fidelity noiseless measurement, which is usu- ally difficult and expensive to obtain. Hence, many recent efforts have been devoted to discover
PDEs from sparse/noisy data (Li et al. 2020a Xu ct al. 2021
Sun et al. 2021 Chen et al. 2021).
METHODOLOGY
3.1 PROBLEM DESCRIPTION
Discovering the explicit form of the governing equation(s) for a given system from the measure- ment data is of critical importance for scientists understand some underexplored processes. To formalize this problem, which is also known as the data-driven equation discovery, let us assume the response of a multi-dimensional spatiotemporal system is described by
(1) where = [x, vl and t denote the spatial and temporal coordinates respectively, denotes the state variable possibly consisting of multiple components (e.g.. u has two components u and v) and the subscript denotes the partial derivatives, e.g., U1 Ou/ Ōx.. F is function parameterizing the system dynamics with possible linear/nonlinear terms (c.g.. u², u, and Vu, where the Nabla operator is defined as (0/ /xz 0/Oy]"). The objective of the data-driven equation discovery is to uncover the closed form of F provided time series measurements on some fixed spatial locations. Although this problem has been studied extensively over the past few years, the combinatorically large search space of the possible equation appears as one of the major obstacles that hinders the discovery of the governing equation from very limited measurements. The existence of noise, as well as the coarse-sampling of the measurement, also adds to the difficulty of this problem. To address these challenges, we proposed a novel chysics-encodee reconstruct the high-fidelity solution from the noisy and LR measurements with the help of prior physics knowledge. We successfully combine this network with the sparse regression for discovering the explicit form of the governing PDE of dynamical systems.
11 = F (x, I, u, u, Uy Ny: Vu,...)
recurrent network that is able to
3.2 PHYSICS-ENCODED NETWORK FOR DATA RECONSTRUCTION
Obtaining the numerical derivatives for constructing the library matrix is crucial to the equation discovery with sparse regression. Due the LR and the noise in measurements, directly applying
3

---

Published as a conference paper at ICLR 2022
Spanse regrestion
ni-block
3b. Physic-bassed FD corv
Mieerty
Damy
1391
31
Cuen
My
amg.nna (B800)) 1155)
Figure I: Discovery of governing PDEs with the proposed Physics-encoded DL framework. 1. Data reconstruction: data-driven model constructed from noisy and LR measurement is used generate HR solution for sparse regression; 2. Sparse regression: STRidge algorithm is used to obtain the sparse coefficient vector E; 3. Coefficients fine tuning: the PeRCNN built based on identified PDE structure is employed to fine-tune the coefficient from the sparse regression.
the traditional finite difference (FD) onto the raw measurements may lead inaccurate derivatives, which would significantly complicate the equation discovery. To handle this issue, we introduce a recurrent network architecture, namely the Physics-encoded Recurrent Convolutional Neural Net- work (PeRCNN) shown in Fig. 11) and (Ib), to establish the data-driven predictive model from LR noisy measurements. PeRCNN consists of a fully convolutional (Conv) network as initial state generator (ISG) and well designed Conv block, namely I1-block (product) for updating the state variables on high-resolution grid recurrently. Note that our network does not require any HR data as the "label" for training, which is fundamentally different from the classic super resolution task.
The network architecture mimics the forward Euler time stepping à (k+1)) = (a + +Fū (6) 0)ot,
where u R denotes the HR prediction on x W grid at 14th time step, ót is the time spacing and 0 is set of trainable variables for approximating F. Since we seek the predictive model for HR solution, the ISG component introduced to generate a HR initial state from the noisy and LR measurement (o) for the recurrent computation. The recurrent 11-block is a major innovation of this network architecture to capture the dynamics of F. As shown in Fig. Ib), the input state variable II-block would firstly be mapped to the hidden space through multiple parallel Conv layers, whose output is then fused via the element-wise product operation. A Conv layer with xx] 1 kernel (Lin et al., 2013) is subsequently used to linearly combine multiple channels into the output (i.e., approximated F). Mathematically, the II-block seeks to approximate the function J via polynomial combination of solution and its spatial derivatives, given by
No
Ni
F(ul =
Je
II
s ú
+ +b²)
(2)
where N₂ and N₁ denote the numbers of channels and parallel Conv layers respectively; @ denotes the Conv operation; (c. I) indicates the Conv filter K. of I-th layer and c-th channel; fe is the weight in 1 x 1 Conv layer while the bias is omitted for simplicity. This multiplicative representation of -T-block promotes the network expressiveness for nonlinear function compared with the additive representation (i.e., weighted sum of multiple output) commonly seen, such as in PDE-Net (Long) ct al., 2018). We demonstrate in Appendix Section A that 11-block is an universal polynomial ap- proximator to the nonlinear function F. In addition, this network architecture features the capability to encode physics knowledge, such as the existing term in the PDE or the known initial and boundary conditions (I/BCs). As shown in Fig. 1bb.. highway to account for existing terms known a priori in the PDEs. In Appendix Section [ we demonstrate that such a highway connection can accelerate the training speed and improve the accuracy of the fitted model. In addition, the knowledge of I/BCs of the system can be encoded into the network through the customized padding Conv operation. For example, the periodic boundary condition (PBC) could be encoded into the model via circular padding on the prediction (c.g.. [1,2,3.4] padded to [4. 144,,,,3,4,11), Note that "physics-encoded" has the meaning of threefold: (1) PDE solution time marching that follows the recurrence of forward Euler integration, (2) encoding priori PDE terms into the Iloblock, and (3) encoding multiplicative polynomial PDE terms into the II-block.
based Conv layer" can be created
This Cony layer with the corresponding FD filter would be freezed throughout the training.
4

---

Published as a conference paper at ICLR 2022
3.3 SPARSE DISCOVERY
Based on the reconstructed HR solution (generated metadata), we further employ the sparse discov- ery algorithm (Rudy al. 2017) to distill the underlying governing equations. The sparse regression method roots on a critical observation that the right hand side (RHS) of Eq. ① for the majority of natural systems consists of only few terms, i.c., the F demonstrates sparsity in the space of pos- sible functions. Given a library of candidate functions of F. we can formulate the discovery of the governing equation as a regression problem. Let us consider the state variable with one single component u E x*, which is defined on spatial locations and within time steps. After flattening the state variable into a column vector U E IR= x1, a library ((U) E R that encompasses a pool of candidate functions (e.g.. linear, nonlinear terms, derivatives, etc.) can be constructed to represent the governing equations of physical systems. Each column in ((U) rep- resents a candidate function in F, such (U) [1, U, U² ..... Up. ...Uy, Uy.... Assuming that (U) has a sufficiently rich column space, the governing equation described by Eq. (I) can be rewritten as a linear system by using a sparse coefficient vector = E Reex1, namely,
U, -
(3)
To compute reasonable E such that its sparsity is satisfied while the regression error is small, the Sequential Threshold Ridge regression (STRidge) algorithm (Rudy al.) (017), among other ef- fective Saassty-promoting methods such as the Iterative Hard Thresholding (IHT) method (Haupt & Nowak 2006, BBlueensth & Davies, 2009), is used in this paper due to its superior performance compared with other sparsity- promoting algorithms, such as LASSO [Tbsshirai] 1996) and Sequen- tially Thresholded Least Squares (STLS) (Brunton et al.] 2016). For a given tolerance that filters the entries of =\t with small value, STRidge can be performed to find the optimal tolerance according to the selection criteria:
obtain sparse representation of F. Iterative search with
E arg min UU, (UU) + |
where |loo is used to measure the sparsity of coefficient vector; ||U, eU)3 denotes the regression error; is the coefficient that balances the sparsity and the regression errors. Since the optimization objective has two components, we apply Pareto analysis to select an appropriate 7 (see Appendix Section H). This procedure to find sparse E is also known as sparse regression.
3.4 HYBRID PEDI. FRAMEWORK FOR EQUATION DISCOVERY
In this part, we present the hybrid/coupled scheme of the proposed PeRCNN and the sparse regres- sion for discovering the governing PDEs. As shown in Fig. T the whole equation discovery process can be divided into three stages, including the metadata reconstruction, the sparse regression for PDE discovery and the fine-tuning of PDE coefficients. The proposed framework enables us to iter- ate over these three stages to keep refining the result until convergence. The purpose of each stage, as well as the technical details, is detailed as follows.
Data reconstruction. Since the experimental measurements are typically sparse and noisy, firstly utilize the PeRCNN introduced Section 3.2 to reconstruct the high-fidelity data in order to obtain accurate derivative terms in the stage of sparse regression. Given the measurement] ū R=(,HH' xW' on a coarse grid (with H' W' resolution) n. time steps, we establish data-driven model that minimizes the misfit error (i.e., mean squared error) between the predictions from the PeRCNN model and the input measurements. specific, the loss function is given by
(()) = MSE (ū(x) - u) + 1/MSE -I('"()()))
(5)
Note that the 1st part in RHS of Eq. (5) is adopted for training the network based on the LR data ü. Since we use & to denote the set of locations (i.e., coarse grid) on which the LR data is collected, (() denotes the mapping of HR prediction i E R x HXW on the coarse grid x. where n, is the number of time steps for reconstructed data and H x W the resolution for the fine grid. Besides, the 2nd part in RHS of Eq. (5) is the regularizer term referring to the discrepancy of measurement- interpolated HR initial state ((ū())) and the predicted HR initial state (0) from ISG, where the
Z herein denotes the I.R measurement while Eq. represents the symbol of state variable.
5

---

Published as a conference paper at ICLR 2022
superscript "0" indicates the first snapshot of the measurement or prediction. The introduction of the initial condition (IC) regularizer is because of its effectiveness in avoiding the model overfitting. Moreover, 17 is the weighting coefficient for balancing these two loss components in the model
training. We first pretrain ISG ensure the accuracy of the HR initial state (i.e., úi (0)) Then the
whole network (ISG and recurrent IT-block) is trained using the loss function of Eq. (
Sparse regression. With the reconstructed high-fidelity (i.e., HR and denoised) solution, we are able to reliably and accurately perform sparse regression for the explicit form (analytical structure) of PDEs. As described in Section given a library of candidate functions ((U), sparse regression seeks to find suitable coefficient vector 3\t that balances model complexity and accuracy. This is realized by solving the optimization problem described by Eq. (4) with the STRidge algorithm. Note that sparse regression is performed on subsampled (U)) (i.e., 10% rows randomly sampled) to avoid a very large library matrix that would lead to extremely slow computation or exceeding computer's memory limit. Similar strategy is adopted in PDE-FIND (Rudy et al. 2017).
Fine-tuning of coefficients. The obtained coefficients from sparse regression may not fully exploit all the available measurement as the regression performed on subsampled HR data. To further improve the result of equation discovery, we present a fine-tuning step to produce the final explicit governing equation. In the fine-tuning step, all the spatiotemporal measurements are used to train recurrent block completely based the identified PDE structure from the sparse regression (see Fig. ([))). The coefficient of each term, which is treated as a trainable variable in the network, can be obtained by minimizing the misfit error between the predictions and measurements. In Appendix we show that fine-tuning can effectively improve the accuracy of the discovered PDE.
Section G
4 EXPERIMENTS
4.1 DATASETS
To examine the effectiveness of the proposed approach, we test on three different datasets (i.e., PDE systems) which cover the 2D Burgers' equation, 2D Lambda-Omes (A-Ω2) and 2D Gray-Scott (GS) ccactionddiffusion (RD) equations. All of them are valuable mathematically and practically, and usually work as the benchmark examples. For instance. 2D Burgers' equation has wide applications in fluid mechanics, acoustics and traffic flow, while the GS RD equation is extensively used to model the process in chemistry and biochemistry (whose response has very complex patterns). Detailed description of each equation is provided in Appendix Section B
4.2 EVALUATION METRICS
We adopt three quantitative metrics to examine the performance of our method for PDE discovery:
Relative l error of coefficients. The relative lg error, defined as E = E B |2/ E l1.. measures the relative distance between the identified coefficient vector 3u and the ground truth Eine However, when the magnitude of coefficients vary significantly (e.g., the GS RD equation). E cannot reflect the result well the small coefficients get overwhelmed. Hence, we introduce the following non con-dimensional measures to gain more insightful evaluation of the results.
Precision and recall. The PDE discovery can also be considered as a binary classification prob- lem (e.g., whether a term exists or not) given candidate library. Therefore, we present the precision and recall to evaluate the performance of the proposed method. The recall measures the percentage of the successfully identified coefficients among the true coefficients, defined as where @ denotes clement-wise product of two vectors" Similarly,
R L @ Emello//|E the precision has the definition of P E O E |//|/aallo-
4.3 EXPERIMENT SETUP
Generation of measurements. The noisy and LR measurements are synthetized through numeri- cal simulations with high-order FD methods. 9-point stencil is used for computing the spatial deriva-
A successful identification occurs when the entries in both identified and true vectors nonzero.

---

Published as a conference paper at ICLR 2022
tives while time marching is performed through Runge- Kutta scheme. For the 2D Burgers' equation
system, we select y/ to be 0.005 (Geneva & Zabaras 2020) and generate the solution on a unit square domain discretized by the 101 101 grid. 200 time steps are simulated for total duration of 0.05
second. The numerical solution is downsampled to measurements as i E R TR10x2251xx3 X-2 RD system, B. Juu and j4, are set be 1, 0.1 and 0.1 respectively. We discretized the 20 x 20 square domain into the 101 x 101 grid. 200 time steps are considered for a physical duration of 2.5 seconds. The measurements are downsampled from the solution be ú E [R40x2x51xx5 For the 2D GS RD system, the solution is generated on the 101 x 101 grid for a unit square domain using the parameters of Pau simulated for the total duration of 400 seconds. The measurements are obtained by downsampling the numerical solution to û E IR160XXX6XX22 Periodic boundary condition is adopted for all three systems. In addition, to mimic the measurement noise in the real world, we add the Gaussian noise of a given level (c.g., 5% and 10%) to the numerical solution before downsampling for the LR data.
For the 21)
10 H4
10 A = 0.06 and f = 0.04. 800 time steps are
Baselines. In the experiments, we compare the performance of our proposed method with the widely used PDE-FIND (Rudy et al., (0177). Different from the settings in (Rudy et al. (0017), we assume only LR measurements are available for computing the derivatives in (U)). Specifically, the FD is used to obtain the derivatives for noiseless data. However, in the case that data is heavily corrupted by noise, the derivatives of the fitted Chebyshev polynomial are used. In addition, we also consider the baselines of the sparse regression coupled with a fully connected neural network (namely FCNN+SR) or PDE-Nef (namely PDE-Net+SR). In these two baselines, the FCNN and PDE-Net (Long et al. 2018) are used respectively to fit the LR measurement and perform inference for HR data while SR is used to discover the PDE. Notably, automatic differentiation (Baydin et al.] 2017) is used in FCNN+SR to compute the partial derivatives in (U). However, like many other methods (Chen et al. 2021 Xu et al. 2021) that utilize the FCNN to fit the sparse data, FCNN+SR lacks the capability of encoding prior physics knowledge. Therefore, the prior knowledge of known diffusion terms would not be utilized in the data reconstruction phase of FCNN+SR.
4.4 RESULTS
2D Burgers' equation. We first test the proposed approach on the 2D Burgers' equation. To recon- struct the high-fidelite data, we use the PeRCNN with 3 parallel Conv layers of 16 channels and filter size of 5. These hyperparameters of network architecture are selected through hold-out validation. The range for selecting the network hyperparameters are given in Appendix Section[[] Furthermore, in this and the following examples, we assume the measurement exhibits the ubiquitous diffusion phenomenon. That said, the diffusion layer would be encoded into the network architecture. Also in the sparse regression stage, the coefficient of diffusion term would be exempted from being filtered. We train the network with Adam optimizer for 15,000 iterations. The learning rate is initialized to be 0.002 and decreases to 97% of the previous for every 200 iterations.
Once the training is completed, the HR solution (i.e., 201 snapshots of 101 X 101) can be inferred from the trained model. Figure 2 a) provides the snapshots of the reconstructed HR data under 5% noise from each method. It can be seen that our method and the FCNN has a much smaller reconstruction error, which would give rise to more accurate derivatives for constructing the library matrix. We also observe that PDE-Net struggles in reconstructing the HR data. Based on our experiments and the claim PDE-Net paper (Long et al. 2018) made, this is because the additive operation limits PDE-Net to only learning the response of linear PDEs. Therefore, unlike our model, PDE-Net lacks the capability to exactly express nonlinear terms like uu, and u²u. To prepare for the subsequent sparse regression, we established a group of 70 candidate functions that consists of polynomial terms {1, u, u, u², ue, v², u², , u²v, wp², v²), derivatives {1, Uzz My: Ca, Uy, Au, Av) and their combinations. The selection of the crucial hyperparameter ? in sparse regression is conducted through Pareto analysis, which is detailed in Appendix Section H In addition to the diffusion terms known a priori, the result of sparse regression indicates the presence of several other terms in F, i.e., S, = {unz, vuy) } and S. = (uvz, vry) respectively. As the last step, the fine-tuning is performed using the network built completed based on the discovered PDE structure in the second
As the original PDE-Net does not account for I.R measurement data, we replace the IT-block in our model with the 5t-block in PDE-Net make it HR predictive model.
7

---

Published as a conference paper at ICLR 2022
Truth
Ours
PDE-Net
FCNN
Poly.
(Ours)
(PDE-Net)
(FCNN)
(Poly)
11
bb)
Figure 2: Snapshots of the state variable one time instance. (a)-(c) represent the 2D Burgers', A-52 and GS RD systems. Six columns starting from the left denote the LR measurement, HR ground truth, reconstructed HR solution from our model, PDE-Net, FCNN and Chebyshev polynomial fit- ting (i.e., PDE-FIND) respectively. The reconstruction error of each model is also provided.
stage (see Fig. D 3b)), which gives us the final discovered equation:
Ut = 5.0113 x 10-3 Au 1.004wug 1.004wuy.
(6)
= 4.9953 x 10 Ao 1.009uc.- 1.002v0y
The precision, recall and relative & error the discovered PDE under various noise levels are provided in Table It can be observed that all methods except PDE-Net+SR perform well when the measurement is noise-free. Furthermore, the performance of baselines deteriorate when the noise level increases to 10% while
approach demonstrates better robustness against the noise. Our approach achieves 5.9 x 10 relative &g error, 100% recall and precision under the 10% noise level. In the Appendix Sections E and F we demonstrate that method could handle even larger noise (up to 30%) and sparser data. We also consider more challenging case (increasing the Reynolds number 500 for the Burgers' equation), in which method still work well for both response reconstruction and PDE discovery (see Appendix Section[]] []. We also show the interpretability of the trained I1-block network for the 2D Burgers' case (see Appendix Section
2D X-2 RD equation. In this example, the recurrent network employed for data reconstruction has parallel Conv layers, 16 channels and ample for training the reconstruction network is adopted here. Once the training is completed, we infer from the network the HR solution at finer spatiotemporal grid. The reconstruction er- of our method and the PDE-FIND is shown in Fig. (bb). The same candidate set in 2D Burgers' example (with 70 candidate functions) is adopted to establish the library matrix (UU). We randomly subsample 10% of the HR points for the sparse regression, whose result (5% noise case) gives the set of terms with nonzero coefficients, i.c., S, (Au, u, u3 u²v, we? u and S, (Av, u, u², u²v,, wv² v²}. With the set of existing terms the governing PDE, the fine- tuning step is performed using all the available measurement. The obtained scalar coefficients from the fine-tuning renders us with the final governing equation, which reads as
kernels. The same procedure previous ex-
Ut =0.096Au + 1.038u 1.048u² 1.004u2v 2.050ur² + 0.998v
04 00.100Av 1.014v 0.998.² 1.025 2.999 1.015
The evaluation metrics computed from the discovered PDE under various noise levels are provided in Table 1 Our approach achieves 5.44 x 10 relative error, 100% recall and 91.6% precision under 10% noise. Our method is also capable of handling much sparser data (Appendix Section
8

---

Published as a conference paper at ICLR 2022
Table Performance comparison between our proposed framework and basclines.
Metries
Relative f2 error /x 10
Precision %%
Recall %%
Cases
Noise level
0%
5%
10%
0%
5%
10%
0%
5%
10%
Ours
0.50
0.54
0.59
100
100
100
100
100
100
Burgers'
FCNN
0.37
0.43
1.34
100
100
75.0
100
100
100
PDE-Net+SR
339.7
442.3
372.0
40.0
25.0
28.5
66.7
50.0
33.3
PDE-FIND
3.32
36.80
45.09
75
100
62.5
100
83.3
83.3
Ours
1.18
2.69
5,44
100
100
91.6
100
100
100
X-Q RD
FCNN+SR
5.19
7.50
15.85
100
85.7
85.7
100
100
100
PDE-Nct+SR
62.23
107.45
104.63
81.82
64.3
50.0
75.0
75.0
50.0
PDE-FIND
1.70
92.52
99.15
100
83.3
77.8
100
62.5
58.3
Ours
1.59
2.85
10.03
100
100
85.7
100
100
85.7
GS RD
FCNN+SR
95.14
143.55
162.98
37.5
33.3
33.3
60.0
57.1
57.1
PDE-Net+SR 204.61
100.00
382,20
30.8
40.0
30.0
57.1
28.6
42.9
PDE-FIND
113.05
89.99
108.1
45.5
50.0
42.9
85.7
57.1
60.0
2D GS RD equation. In the last example, we assume the measurements are some extremely LR (i.e.. 26 x 26) snapshots (see measurements in Fig. [c)). To remedy the lack of resolution in spatial dimension, we further introduce the assumption that reaction term in the form of poly- nomial after reviewing the existing literatures. As the first step, we employ the network with 3 parallel Conv layers, channels and filter size of to reconstruct the data. From the reconstruc- tion model, infer the HR (i.e., 101 101) solution at finer time instances. Snapshots of HR solution in Fig. ([c) show our method able to restore the high-fidelity data from the LR and noisy measurement very well. With the HR solution, library of polynomials up to the third degree @(u, v) = [1, u, v, u², uv, v², u² u²v, un?, x0 is constructed for sparse discovery. We subsample 10% HR data for the sparse regression, which gives us the remaining terms S,, (Au, 1, u, uv²) and S. = (Au, v, w²².. With the existing terms in the PDE, we perform the fine-tuning using the completely physics-basee based Conv block. The final obtained PDEs the case of 5% noise are
u =2.001 X 10-5 Au 1.003wr2 0.04008u + 0.04008,
1/2 =5.04 x 10 6 Au v + 1.009wx² - 0. 10070.
As presented in Table our proposed approach performs well on discovering the PDE as result of fully utilizing the prior physics knowledge and the powerful expressiveness of the model, while PDE-FIND, PDE-Net and FCNN+SR struggle due to the extremely LR measurements.
5 CONCLUSION
We propose a hybrid computational framework for discovering nonlinear PDE systems from sparse (LR) and noisy measurement data, which couples the novel network architecture PeRCNN for the data-driven modeling of dynamical systems and the sparse regression for distilling the dominant candidate functions and coefficients. First of all, one major innovation of our framework scien- tific discovery comes from the nonlinearity achieved by the element-wise product operation and its powerful expressiveness (see Appendix Section A) for the nonlinear function F-)) in PDEs. Be- sides, our network also features the capability to encode prior physics knowledge via highway Conv layer that helps to overcome the challenges brought by the scarcity and noise of the mea- surements. Secondly, we successfully marry PeRCNN to the sparse regression algorithm to solve the crucial equation discovery issues. The coupled scheme enables us to iteratively optimize the network parameters, and fine-tune the discovered PDE structures and coefficients. Through the numerical validation, we demonstrate the superior reconstruction capability and excellent discov- ery accuracy of our proposed framework against three baseline approaches. Overall, provide an effective, interpretable and flexible approach to accurately and reliably discover the underlying physical laws from the imperfect and coarse-meshed measurements.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
The work is supported in part by the Beijing Outstanding Young Scientist Program (No. MUIZYYH012019100022000 as well as the Intelligent Social Governance Platform, Major Innovation Planning Interdisciplinary Platform for the "Double-Firs" Class" Initiative, Renmin University of China.
REFERENCES
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven dis- cretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116631)15344-15549 2019.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.0126. 2018.
Atilim Günes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Re- search, ((11:5559--6337 2017.
Filipe de Avila Belbute-Peres, Thomas Economon, and Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In International Conference on Ma- chine Learning, pp. 2402-2411, PMLR, 2020.
Thomas Blumensath and Mike Davies. Iterative hard thresholding for compressed sensing. Ap- plied and computational harmonic analysis, 27(3):265-274, 2009.
Josh Bongard and Hod Lipson. Automated reverse engincering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104(24):9943-992) 2007.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113(15):3932-333) 2016.
Jared L Callaham, Kazuki Maeda, and Steven L Brunton. Robust flow reconstruction from limited measurements via sparse representation. Physical Review Fluids, 101007 2019.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature Communications, 12:6136, 2021.
Miles D. Cranmer, Alvaro Sanchez-Gonzale Peter W. Battaglia, Rui Xu, Kyle Cranmer, David N. Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In NeurIPS, 2020.
Saso Dzeroski and Ljupco Todorovski. Discovering dynamics: from inductive logic programming to machine discovery. Journal of Intelligent Information Systems, 4(1):89-108, 1995.
N Benjamin Erichson, Lionel Mathelin, Zhewei Yao, Steven L Brunton, Michael W Mahoney, and Nathan Kutz. Shallow neural networks for fluid flow reconstruction with limited sensors. Pro- ceedings of the Royal Society A, 477622238)200200977 2020.
Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A Tchelepi, Philip Marcus, Mr Prabhat, Anima Anandkumar, et al. Meshfreeflownet: a physics- constrained deep continuous space-time prace-time super-resolution framework. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. IEEE, 2020.
10

---

Published as a conference paper at ICLR 2022
Mojtaba F Fathi, Ali Bakhshinejad, Ahmadreza Baghaie, David Saloner, Raphael H Sacho, Vitaliy L Rayz, and Roshan M D' Souza. Denoising and spatial resolution enhancement of 4d flow mri using proper orthogonal decomposition and lasso regularization. Computerized Medical Imaging and Graphics, 70: 16-112, 2018.
Kai Fukami, Koji Fukagata, and Kunihiko Taira. Machine-leaming-bassd spatio-temporal super resolution reconstruction of turbulent flows. Journal of Fluid Mechanics, 909, 2021.
Han Gao, Luning Sun, and Kan-Xun Wang. Phygeonet: physics-informe geometry-adaptive convo- lutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428: 10079, 2021a.
Han Gao, Luning Sun, and Jia Wang. Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels. Physics of Fluids, 33(7):073603, 2021b.
Han Gao, Matthew Zahr, and Jian-Xun Wang. Physics-informed graph neural galerkin networks:
A unified framework for solving ode-governed forward and inverse problems. arXiv preprint arXiv:2107.12146 2021c.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403:109056, 2020.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474-1 11484, 2020.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115((4):8505-855) 8510, 2018.
Jarvis Haupt and Robert Nowak. Signal reconstruction from noisy random projections. IEEE Trans- actions on Information Theory, 5209):403644048 2006.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang- Physics-informed machine learning. Nature Reviews Physics, 3(6):222-444 4204242224422 2021.
Samuel Kim, Peter Y Lu, Srijon Mukherjee, Michael Gilbert, Li Jing. Vladimir Čeperić, and Marin Soljačić. Integration of neural network-based symbolic regression deep learning for scientific discovery. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 100.000099 2016.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine soaming-acceleratn computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
Isaac E Lagaris, Aristidis Likas, and Dimitrios Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 955:987711000, 1998.
Hyuk Lee and In Seok Kang. Neural algorithm for solving differential equations. Journal of Com- putational Physics, 9101:1100-13,, 10-131, 1990.
Jun Li, Gan Sun, Guoshuai Zhao, and Li-Wei H. Lehman. Robust low-rank discovery of data-driven partial differential equations. In AAAI, pp. 767-774. AAAI Press, 2020a.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint 1rXiv:1810.05555 2018.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint 100.0:20 2020b.

---

Published as a conference paper at ICLR 2022
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Bo Liu, Jiupeng Tang, Haibo Huang, and Xi-Yun Lu. Deep learning methods for super-resolution reconstruction of turbulent flows. Physics of Fluids, 32(2):025105, 2020.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data, In International Conference on Machine Learning, pp. 3208-3216 PMLR, 2018.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 1999:08925,, 2019.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence. 3(3):218-229, 2021.
Ravi G Patel, Nathaniel A Trask, Mitchell A Wood, and Eric C Cyr. A physics-informed operator re- gression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 373:1 13500, 2021.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equa- tions. The Journal of Machine Learning Research, 19(1):932-955, 2018.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Chengping Rao, Hao Sun, and Yang Liu. Embedding physics learn spatiotemporal dynamics from sparse data. arXiv preprint arXiv:2106.04781, 2021a.
Chengping Rao, Hao Sun, and Yang Liu. Physics-informed deep learning for computational elasto- dynamics without labeled data. Journal of Engineering Mechanics, 147(6):04021043 2021b.
Pu Ren, Chengping Rao, Yang Liu, Jianxun Wang, and Hao Sun. Phyernet: Physics- informed convolutional-recurcee network for solving spatiotemporal pdes. arXiv preprint arXiv:2106.144 14103, 2021.
Samuel H Rudy, Steven L. Brunton, Joshua L Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4):e1602614 2017.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning, pp. 4442-4450 PMLR, 2018.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- on Machine Learning, pp. 8459 8468. PMLR, 2020a.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 8459-8466 PMLR, 2020b.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81-85,, 2009.
Karen Stengel, Andrew Glaws, Dylan Hettinger, and Ryan N King. Adversarial super-resolution of climatological wind and solar data. Proceedings of the National Academy of Sciences, 117(29): 16805-16815, 2020.
Fangzheng Sun, Yang Liu, and Hao Sun. Physics-informed spline learning for nonlinear dynamics discovery. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI-2I), pp. 2054-2061, 2021.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), (8(1):267-288. 1996.
12

---

Published as a conference paper at ICLR 2022
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference on Learning Representations, 2019.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1457-1466, 2020.
Hao Xu, Dongxiao Zhang, and Junsheng Zeng. Deep-learning of parametric partial differential equations from sparse and noisy data. Physics of Fluids, 33(3):037132, 2021.
Bing Yu al. The deep ritz method: deep baaeingg-basedd numerical algorithm for solving varia-
tional problems. arXiv preprint arXiv:1710.00211, 2017.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018.
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data, Journal of Computational Physics, 394:56-88 81, 2019.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A [I-BLOCK AS UNIVERSAL POLYNOMIAL APPROXIMATOR
In the proposed PeRCNN, 11-block acts as an universal polynomial approximator to unknown non- linear functions while the physics-basse highway Conv layer (i.e., with FD stencil as the Conv filter) accounts for the prior knowledge on the governing equation. Notably, the II-block achieves its non- linearity through the element-wise product operation (see Eq. (2) instead of what is widely adopted by the traditional Conv network, in which nonlinear layers are sequentially interwined with linear
layers. Compared with the additive form representation F(u)
Σ fi (Kis @ u) seen in
0C++JEN
related work (Long et al. 2018 Guen & Thome 2020), this multiplicative representation of II-block promotes the network expressiveness for nonlinear functions. To support this claim, we need to rec- ognize that II-block roots on the numerical differentiation (i.e., forward Euler time stepping), which can guarantee the convergence of the solution under two conditions: (i) the RHS F of Eq. computed accurately; and (ii) the time spacing 5l is sufficiently small. As the second condition can be satisfied by selecting appropriate òt, we would show that, by construction, our TT-block can approximate any nonlinear polynomial involving spatial derivatives, such as Wlg + tuyy and wAu. This can be easily argued by proving that each monomial (e.g.. uug, o'ly, etc.) can be approximated by multiplying the output of given number of Conv layers. To this end, we provided the following lemma and its proof reproduced from (Long et al. 2019).
be
Lemma I:
By construction, convolutional filter K. with given size can approximate any differen- tial operator with prescribed order of accuracy.
Proof I: Without loss of generality, let us consider a bivariate differential operator 2(-) w.r.t vari- ables x and y, we have
gitiu Ooayy y (x.w)
kik2 il!!
[(u)
Σ
Kiki 12]
da'dy' + O(dxx N-1 +10y1
k Rog
1.j=0
(9)
Σ
Kik,, (s]u(( + kns dx, y + k20z) + O(66x|N-1 1811)
Ko (18x1| 11
where N is the size of the filter. The weight matrix KT[., is indexed by L and l2. Letting the filter's entry KCik,, k2] be the corresponding Taylor series coefficient, we can see the approximation error is bounded by ((88x|~-1 11 + N-y).
Therefore, compared with the traditional black-box models (e.g., deep neural networks) for repre- senting nonlinear functions, II-block possesses two main advantages:
The nonlinear function F in the form of multivariate polynomial (e.g., u vu+u²e) covers a wide range of well-known dynamical systems, such as Navier- reaction-diffusion (RD), Lorenz equations, to name only a few. Since the spatial derivatives can be computed by Conv kernels, a II-block with parallel Conv layers of appropriate filter size is able to represent polynomial up to the th order.
The II-block is flexible in representing the nonlinear function F. For example, I[-block with parallel layers of appropriate filter size ensembles a family of polynomials up to the 2nd order (e.g., u, Au. ue, u Vu), with no need to explicitly define the basis functions.
B DETAILED DESCRIPTION OF THE CONSIDERED PDES
The first 2D Burgers' equation studied in this paper has the form of
+ Vu AAu
(10)
14

---

Published as a conference paper at ICLR 2022
where V is the Nabla operator, A is the Laplacian operator; [u, 2)" is the fluid velocities and y/ is the viscosity coefficient. The second and third RD equation system can both be described by
Ur DÂu + R(u)
(11)
where = [u, w|" is the concentration vector; D [Mr. Hbl the diagonal diffusion coeffi- cient matrix; R(u) denotes the reaction vector. For the X-22 RD system, the reaction vector has the form of R(u) where 8 is reaction parameter, For the GS RD system, the nonlinear reaction vector reads R(u) = [-ur² + f(1 - u), wv²² - (f + k)ul² where x and f denote the kill and feed rate re- spectively. The dataset and training script for each case considered in this paper can be found in https://github. www.huu...oo/aaoooce/eee http///acp//iiieeereeeeeeeeeeeee wwweeweer-----wttttee witth Noisy- Ser--Scaree-aaae
[[1 2² 2d )u + (u2 + ye, {((2 + v²)u + (1 u² )u]=
C HYPERPARAMETHR SELECTION OF PERCNN IN DATA RECONSTRUCTION
The selection of the hyperparameters of PeRCNN for data reconstruction is performed through the hold-out validation. Among the entire measurement, 10% data is split as the validation dataset. The range of hyperparameters we considered summarized in Table C.I The value in the parenthesis denotes the final parameter adopted in the data reconstruction.
Table C.I: Range of hyperparameters for the data reconstruction network.
Dataset
Kernel size
# layers
# channels
Learning rate
Regularizer weight v
2D Burgers'
(5)
2n4 (3)
4~16 (16)
0.001~0.01 (0.002)
0.001~1 (1)
2D X-Ω
I~5 (5)
2-4 (3)
4~16 (16)
0.001~0.01 (0.002)
0.001~1 (0.01)
2D GS
1~5 (1)
2n4 (3)
4~166 (8)
0.001~0.01 (0.001)
0.001~1 (0.005)
D ENCODE PHYSICS KNOWLEDGE TO ENHANCE THE MODEL
As introduced in Section 3.2 one salient characteristic of the proposed network is the capability to encode prior physics knowledge. To examine how the encoded physics enhances the accuracy of the data-driven model, here we use the 2D Burgers' equation system a testbed. In the experiment, we consider two different cases: (1) no prior knowledge is available, and (2) the diffusion term (Au) is known to exist in F. The setting in the second case leads to additional thysics-based Conv layer in PeRCNN to account for the diffusion term. Apart from this, other hyperparameters are kept same in the experiment.
Figure D.I shows the learning curves of these two data-driven models. It can be observed that the network encoded with the diffusion term exhibits faster convergence and better accuracy compared with the competitor. As a result of encoding prior physics knowledge, the PeRCNN does not need to learn diffusion operator through the back propagation of error. That said, incorporating available physics can facilitate the learning efficiency and accuracy of the data-driven model.
E TOLERANCE TO LARGER NOISE LEVEL
To examine the scalability of our method with regard the Gaussian noise level, we performed additional experiments with the noise level of 20% and 30% for all three cases. The same settings from Section 4 is adopted in these experiments. The performance of our method on various noise levels is summarized in Table E.T We observed that larger noise affects the accuracy of our method differently. For the 2D GS RD equation, the accuracy of our method deteriorates drastically as the noise level increases. That is because the measurement data used in this example has relatively low resolution (i.c., 26 x 26). Even worse, the phenomenon 2D GS RD system is characterized with local patterns and sharp gradients (see Fig. 2 Increasing the Gaussian noise would further blur the spatial information in the measurement data. However, for the case of 2D X-Q RD and Burgers' equation where the measurement data has decent resolution (i.e. 51 x 51) and smoother spatial patterns, our method stays competitive even for 30% Gaussian noise. Overall, there is a trade-off between the sparsity and noise level of the measurement data.
15

---

Published as a conference paper at ICLR 2022
10
Training (w/ diffusion)
Trainin(w// diffusion)
Validation (w/ diffusion))
Validation diffusion)
Loss
10-2
3000
6000
9000
12000
15000
Iterations
Figure D.I: Learning curve of the data-driven model with (w/) and without (w/o) encoded diffusion layer for 15,000 iterations.
Table E.1: Performance of the proposed method on larger noise level.
Cases
Noisc level /%
Precision /%
Recall /%
Relative &a error /x 1
0
100
100
0.50
100
100
0.54
Burgers'
100
100
0,59
20
75.0
100
7.33
30
75.0
100
9.10
100
100
1.18
100
100
2.69
X-Ω RD
10
91.6
100
5,44
91.6
100
7.65
91.6
15.89
100
100
1.59
100
100
2.85
GS RD
10
85.7
85.7
10.03
20
46.2
85.7
86.05
30
33.3
37.5
238.31
F MEASUREMENT DATA SPARSITY
Our method assumes the measurement data is collected on fixed coarse sensor grid (e.g., 21 x 21 sensors), captured either by point-wise sensor units by imaging techniques. Although the ground truth data is of high fidelity, our measurement data is heavily ppaiioemporallyydoowssaaapp as a result, remains sparse (e.g., finite number of LR snapshots). As for the scalability with data sparsity, we performed numerical experiments on the 2D GS and X-Ω RD equations using various spatial resolutions, namely, 51 x 51, 26 x 26, 21 x and 11 11. The observation is that the tolerable sparsity of method depends the spatial pattern of the system. For example, in the 2D GS RD example, the lowest resolution of data to guarantee the discoverability of the governing PDE is 26 26 since the very complex maze-like pattern (see Fig. (2) is in relatively fine scale. However, since the solution in the X-Ω RD example is much smoother and periodic (see Fig. 2b), method is able discover major portion of the PDE (i.e., precision 90.9%, recall 83.3%) with the spatial resolution as low as 11 x 11 (a very sparse data scenario).
and,
G IMPORTANCE OF THE COEFFICIENT FINE TUNING
Our proposed approach for discovering PDEs consists of three stages including the data reconstruc- tion, the sparse regression and the fine-tuning of coefficients. In fact, the sparse regression has
16

---

Published as a conference paper at ICLR 2022
already been able to determine the explicit form of the discovered PDE. Then natural question would be why the last fine-tuning step is necessary. As its name indicates, the fine-tuning step is able to further improve the aceuracy of the discovered PDE, for mainly two reasons: (i) The sparse regression step gives an explicit PDE based on the primitive assumption a sparse E. However, the discovered PDE can be further considered as the latest assumption, based on which completely physics-based recurrent network can be established. In the fine-tuning step, the functional form of the PDE unchanged while its scalar coefficients are optimized. That said, the fine-tuning step is an optimization process on much smaller parameter space. (ii) The sparse regression is performed on subsampled nigh-dimensional spatiotemporal data while the fine-tuning step could utilize all the available measurement through the recurrent network.
The above argument provides some intuitions of the coefficient fine-tuning step. To further justify it, we compute the relative & error of the coefficient vector before/after the fine-tuning step, which is given in Table It can be seen that fine-tuning step improves the accuracy of the discovered PDE consistently for different noise level. Interestingly, we observe that for the 2D GS and X-Ω RD case under 10% noise, the improvement brought by the fine tuning step is not as evident as that of the other cases. That is because, in these two cases, the sparse regression does not give the correct form of the PDE. In other words, the fine-tuning is performed based on an assumption that deviates from the fact. The dependency on the sparse regression result also implies one limitation of our approach.
Table G.I: 1: Relative & error (unit: 10 10-3) of the coefficient vector before and after the fine tuning.
Noise level
Case
Stage
0%
5%
10%
Before
0.93
1.57
2.18
Burgers'
After
0.50
0.54
0.59
Before
1.52
3.33
6,65
A-Q RD
After
1.18
2.69
5.44
Before
6.40
9.84
12.33
GS RD
After
1.59
2.85
10.03
H PARETO ANALYSIS FOR HYPERPARAMETER SELECTION
sparse regression is formulated an optimization problem described by Eq. (4). The aggregated optimization objective consists of two components, ||U, {U)) and | to ensure the accuracy of the regression while keeping the model parsimonious. This trade-off can be balanced by selecting an appropriate weighting coefficient 7 in the aggregated objective function. Pareto analysis so far considered as the most effective way for selecting this hyperparameter 7, To ensure the scale of two components aligned, we introduce an equivalent hyperparameter λ as the nondimensional weighting coefficient. The and y are interconvertible with the definition y = U, @UU)El|2 where E is the Least Squares solution the linear system.
As introduced in Section 3.3
In the Pareto analysis, we would sample some points of k from [10-2, 20]. For each of the point, we would pass it as the argument for the sparse regression routine, which would return the optimal coefficient vector E The corresponding regression error and to penalty can be computed for Pareto analysis. Figure H.I provides the results of Pareto analysis for 2D X-Ω equation system (10% noise case). From the subfigure on the left, we can identify the Pareto front that corresponds to the optimal region of k. Pareto analyses such are performed each case find the suitable weighting coefficient i (or equivalently 7). The value of 6 employed in each case is summarized in Table H.]
However, readers should note that the Pareto front may not always be obvious identify. In many scenarios, trade-off between the model sparsity and accuracy must be done with manual selection, which still appears be more art than science.
17

---

Published as a conference paper at ICLR 2022
(a)
100
Regression ercot
4 pebalty
0.0015
103 7
0.0010
0.0010
Optimal region
0.0005
0.0005
0.0000
0.0000
40
[
(b)
Regression
0.0015
0.00015
4 penalty
Ml
[02 Zile
Optimal region
0,0010
0.00110
0.0005
0.0005
10
30
10
10
10
Figure H.I: Pareto analysis for selecting the weighting coefficient &s (or 7) in 2D X-Ω GS system (10% noise level). (a) and (b) correspond to the Fa and Fu respectively. The Pareto front that represents the optimal is marked with the red star.
Table H.1: The coefficient employed in sparse regression.
Burgers'
À-Ω RD
GS RD
Method
0%
5%
10%
0%
5%
10%
0%
5%
10%
Ours
0.1
PDE-FIND
0.1
0.05
0.1
I DISCOVERY OF BURGERS' EQUATION UNDER HIGHER REYNOLDS NUMBER
In this part, we examine the performance of our method on Burgers' equation when Reynolds num- ber grows. Higher Reynolds number would complicate the equation discovery problem due the intense convective phenomena (or even chaos). We increase the Reynolds number of the problem to 500 by setting the 0.002 both the average initial velocity magnitude and characteristic length is I. 10% Gaussian noise is adopted in this example while the remaining setting follows the Section One snapshot of the reconstructed HR data, altogether with the ground truth, presented in Fig. It can be observed that the reconstructed HR agree well with the ground truth. Then the constructed HR data used in sparse regression to obtain the structure of the governing PDE, based which the PeRCNN is built for coefficient fine tuning. The discovered PDEs after the coefficient fine tuning are
U, = 1.9854 x 10 A 1.0072mm, 1 1.006vuy:
(12)
U; = 1.9409 x 10 10-3 AA 0.9953uv, - 1.008cu,
which match the ground truth correctly.
J INTERPRET THE LEARNED MODEL
One major drawback of the traditional deep neural network is the lack of interpretability as the net- work's output is usually expressed prolonged nested function. However, since each channel of
18

---

Published as a conference paper at ICLR 2022
Meas.
Truth
Ours
Err (Ours)
0.3
0.1
u
0.0
0.9
0.1
-0.6
0.0
Figure I.1: Snapshots of the state variable components (i.e., u and v) for Burgers' equation system with Reynolds number equals to 500.
the input to II-block (i.e., à ( corresponds to a state variable component (i.e., [u, vl). the multi- plicative form of II-block makes it possible extract (or interpret) the explicit form of learned JF from the learned weights and biases via symbolic computations. This is notable advantage of the proposed IT-block as an analytical expression would help researcher to disentangle the underlying physics make inference different cases (e.g., different I/BCs).
Here simple experiment is conducted on the 2D Burgers' case. The network employed in the experiment has two Conv layers with two channels. Two channels of the first Conv layer are asso- ciated with 0/Oz and 0/0y respectively, by fixing the filters with the corresponding FD stencils. The remaining settings kept the same the 2D Burgers' case in Section 4.4 The interpreted expression from the recurrent block (i.e., II-block and the highway diffusion layer) model is
00.055 Au (1.07 0.0065e 0.17) (0.00, 1.01e + 0.17) 0.053
(13)
0.0051 0.82v, (1.22 + 0.0078e 0.18) 0.910,(0.0063u 1.08v 0.17) + 0.058
It is seen that the equivalent expression of the learned model is close to the genuine governing PDEs, which helps explain the extraordinary expressiveness of our model. Although the selection of differential operators to be frozen is crucial for identifying the genuine form of the F, this example demonstrates the better interpretability of PeRCNN over "black-box" models. Similarly, with the trained model from (ection[444|p for 2D GS RD equation which has three Conv layers and 1 X I filters, we could extract a third degree polynomial for the reaction term in
19


Reviews:
[{"review_id": "zY2cLXg58P4", "reviewer": "Reviewer_yjyY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments:\n\n-) The written English language needs to be improved -- the paper is reasonably written from a technical perspective but \ngrammar is not on par.\n\n-) In Section 4, how is the ground truth obtained?\n\n-) One thing I find missing is wall-clock times. I can see that for the experiments and parameters reported in Section 4, the \nproposed algorithm returns better qualitative results. Nonetheless, how much faster/slower is the new method versus the \ncompetition?\n\n-) My main concern is that the sparse regression step can be very slow due to the very large library \\Theta(U). Isn't a pre-processing \nstep like PCA needed in practice? The reason I mention this is because several columns of this matrix are not contributing much.\n\n-) In terms of novelty, the sparse regression step is fairly well-known. I know of some work done on reconstructing noisy samples \n(the authors cite a good part of the related literature on the topic) but how exactly step 1 is really different from previous work? \nThe authors discuss the topic but it's not really clear.\n\n-) Unfortunately, the fine tuning step depends on that of sparse linear regression, i.e., the correct PDE form need be determined. \nThe authors mention this themselves, but any guidance on whether the entire mechanism need be restarted (with different parameters) \nor some of the work can be salvaged would be interesting.\n\nMinor comments:\n\n-) \"Recently, a ground-breaking work of sparse identification of nonlinear Dynamics (SINDy)...\" Please capitalize as needed.\n-) \"Furthermore, the introduction of spatial derivative terms and improved the sparsity-promoting algorithm of sequential threshold ridge regression...\" This phrase doesn't make sense.\n-) \"...many works attempt to relieve the pain...\" Please re-phrase. ", "summary_of_the_paper": "The focus of this paper is to develop a learning framework to discover spatiotemporal PDEs from scarce and noisy data. \nThe authors suggest devise a deep convolutional-recurrent network which encodes prior physics knowledge followed by \nsparse regression with reconstructed data to identify the analytical form of the governing PDEs. The framework is validated \non three high-dimensional PDE systems where superiority over baselines is demonstrated.", "summary_of_the_review": "Overally, the proposed algorithm is discussed at a good abstract level, and experiments on a few small datasets are provided. \nSee my comments above for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "o8ffGOC71o", "reviewer": "Reviewer_t1Ji", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1] Procedure is well-defined/described*; Clean figure 2. The method makes sense to me.\n\nSuggestions\n\n\n1] At first reading, I had a hard time parsing the 2nd contribution bullet point. I was thinking \"Is sparse regression the contribution? The inheritance?  Fine-tuning? Or the actual progression through these stages?\" After reading, I see what you mean, although instead of 'inheritance' maybe you should say simply how you put it later in the paper \"In the fine-tuning step... measurements are used to train a recurrent block completely based on the identified PDE structure from the sparse regression.\"\n* The other thing that took me a second to understand was the recurrent block fine-tuning. I think I understand now that you are scrolling through time computing u_k+1 from u_k using the block from Fig 3b. It may be useful to put the Fine-Tuning loss function -- is it the full vector ||u - U(initial)|| ? Or tuning at each time step before going to the next?\n\n2] I suggest to boldface the winning values in each column of Table 1.\n\n3] The following notes are why I scored a value of 3 on \"correctness of claims\" and \"empirical significance\":\nWhen I see such a huge improvement over the baseline, ...\na) I wonder if there is a better comparator worth considering? Unfortunately I am not well versed enough to suggest something specific.\nb) I would like to see a progression of difficulty starting from when the methods do approximately the same (e.g., HR non-scarce data) to when a winner emerges (LR + scarce) ", "summary_of_the_paper": "Problem: data-driven PDE discovery methods are not robus to low-quality measurement data.\n\nSolution: A novel architecture that can encode prior knowledge (known terms, PDE structure, boundary conditions), and sparse regression procedure that is hypothesized to be more robust to scarce and noisy data scenarios.\n\nOther Contributions:\n* proof that proposed Pi-block is a universal polynomial approximator. This is why I scored a \"4\" on technical contributions.", "summary_of_the_review": "The paper presents a clearly described and sensible approach to PDE discovery. The main problem they seek to solve is robustness to data quality and scarcity, which they address with a combination of sparse regularization, fine-tuning, and prior knowledge encoding architecture. I am not extremely familiar with the field but I believe, especially the latter technique, is quite novel. My only real complaint is that only simulation data was used, and it's not \"stress tested\" in the experiments or against competitors in terms of practical/empirical significance (and hence fully supporting the claims of robustness). Note: I scored myself as a 3 because I'm not that familiar with the related literature.\n\n\nEdit: My confidence on this topic is low because I do not know the literature a priori. However, I am not convinced by Reviewers x24u, zdB8 that the Pi-block is redundant with PDE-Net (to me it seems more efficient and sparse, at least), and I call into question Reviewer zdB8's claim that this is a \"more complex\" system than DLGA-PDE. All in all the other reviewers have shaken my confidence in the empirical results and rigor is methodology descriptions, but not as much in the novelty. I will lower my score to 6 to reflect this.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "baNbq9SR5-D", "reviewer": "Reviewer_x24u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors make substantial clarification and revisions upon the reviews. Especially, the product operator in the proposed block is indeed expanding PDE-net's capability. Based on these, I raise my score to 5, although still not meeting the acceptance bar. The authors may need more time to revise the manuscript but it has been better than before. The main concern is experiments, which need more time. The presentation also needs time for reconsideration and refining. I raise my confidence from 4 to 5 based on the discussion with the authors.\n\n-------------------------------------------------------------------------\n\n○ Strengths:\n1. The ability to reconstruct higher-quality data by available physics data.\n2. Promising results over PDE-FIND (Rudy, et al. 2017.) on the selected equations.\n3. A very thorough review of prior works.\n\n○ Weaknesses:\n1. The novelty is limited. The authors propose to reconstruct higher-fidelity data via a recurrent convolutional network, which is straightforward but not very novel. Many image resolution recovery papers have used similar techniques of Deconv to increase resolution. In Eq. (2), Conv filters are used to process data, while there is no indication of how physics information is encoded. In Eq.(5), the loss is computed by both u and u0. It is not sure how and what physics is encoded into your system as claimed in the introduction. Could you explain?\n2. Confusing presentation. Many notations are not clearly stated. For example, on Page 5, the authors write both x˜ and u˜ as measurements, which is confusing. On page 3, x and t are not introduced. From the paper, readers can hardly interpret their meanings. What is u_y if y is not an input variable in Function F? It is confusing to put u_{k+1} and u_t together, as the first one means time step while the latter one means time derivative. Moreover, what is the meaning of the dimensionality of u (1xn)? Which one is the spatial dimension and which one is the temporal dimension? The definitions are not included or are vague. \n3. Only one baseline published in 2017 is compared in the experimental section, while many papers within the recent three years also claim to improve the robustness of PDE-FIND and solve the low-quality data problem, such as [1-4]. Thus, this paper’s improvement is not persuasive enough without comparisons to state-of-the-art models. \n4. This method solely discusses the discovery of PDEs with constant coefficients and discrete data and the selected three equations in the experimental section are quite simple compared to equations used in prior works, whereas many papers in recent years have moved forward to the discovery of more complex PDEs with continuous mesh-free data. Why not experiment on more complex PDEs like prior works do to adapt the real-world scenarios, such as the 2D Navier-Stokes equation? The influence of this work is limited if only discrete data and constant coefficients on simple PDEs are discussed. \n5. There is no demonstration of real-world data. The data used in this work is solely based on mesh-based simulation, while most real data do not have high-fidelity at all mesh-grid points, which means the data is not only noisy but also sparse. The authors only discuss the robustness against noise. How well does the method scale with sparsity? \n6. The noise range tested in this paper is limited compared to previous work. The authors show that their method is good with noise up to 10%, however, most prior works can also handle 10%. I would be interested to know whether the method can handle larger noise to compete with prior works.\n\n[1] Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. arXiv preprint arXiv:2005.03448, 2020.\n[2] Jun Li, Gan Sun, Guoshuai Zhao, and Li-Wei H. Lehman. Robust low-rank discovery of data-driven partial differential equations. In AAAI, pp. 767–774. AAAI Press, 2020a. \n[3] Hao Xu, Dongxiao Zhang, and Junsheng Zeng. Deep-learning of parametric partial differential equations from sparse and noisy data. Physics of Fluids, 33(3):037132, 2021.\n[4] Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time PDEs from sparse data with graph neural networks. In International Conference on Learning Representations, 2021.\n", "summary_of_the_paper": "The authors propose a physics-encoded discrete learning framework to improve the robustness of the PDE-FIND algorithm. The physics knowledge such as known terms and initial conditions can be used to reconstruct higher-resolution data. Then, sparse regression is performed with fine-tuning of coefficients to finally determine the discovered PDE.", "summary_of_the_review": "The paper is technically correct but lacks novelty and clarity. The experimental demonstration is not sufficient to support that it can also work for state-of-the-art PDE discovery models and for more complex PDEs. The reported robustness is also not exceeding the state-of-the-art models. Besides, the presentation and notation definitions can be improved. Based on these observations of the submitted paper, I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VOJ2cJ3CNpg", "reviewer": "Reviewer_zdB8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Updates after discussions\nMy concerns are partly resolved and I raise my score to 5. Detailed can be found in the discussion threads below.\n\nGenerally this paper is clearly-written and the PDE discovery problem is of great importance in the pioeering research fields. This paper uses some deep cov-net and sparse learning techniques to learn the promising results. \n\n### Main concerns\nThe main concern of mine is the novelty and experiments.\n\n* PeRCNN is proposed in the paper to generate HR data from LR data. However, this method is similar to the existing idea of directly building a neural network to generate meta data, such as DL-PDE[1]. So how much does a more complex network model (i.e., PeRCNN) improve the performance? In the experiment corresponding to Figure 2 and Table 1, the authors should consider comparing the performance of PeRCNN with the method of using a simple fully connected neural network to generate HR data. In fact, many algorithms based on sparse regression or genetic algorithms are already optimized for LR data in PDE discovery. \n\n* The authors are suggested to compare more existing studies to show the advantage of the proposed method. Although PDE-FIND is classic and effective, it is not robust enough to noise and requires that the candidate set contain the correct solution. Therefore, there have been many new attempts in recent years, and the author should compare at least some of these studies. For example, the DLGA that also uses NN to reduce the interference of noise on the model [1], the EPDE that uses genetic algorithm to expand the candidate set [2], the AI Feynman that based on symbolic regression [3], and the SGA that can mine open-form equations form data without prior knowledge of the function terms [4], and the KO-PDE that can discover highly nonlinear coefficients of the PDEs [5], etc. Besides, authors could discuss the difference between the Π-block and PDEnet/PDEnet2.0 [6, 7] in the data reconstruction process. \n\n\n* When faced with noisy data, there are two common ways to calculate the gradient/function term: i). Use neural networks to generate meta data, similar to the PeRCNN, and then use the difference method to calculate the gradients.ii). Directly use the automatic differentiation process of the neural network to generate the gradients (e.g. Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems). Both of these ideas are much better than directly differencing the noisy data. The authors should compare more models in the experiments, including the several studies mentioned at the end of the section 2 of this paper, and some other studies dedicated to sparse and noisy data. It is not fair or sufficient to only compare PDE-FIND with the proposed model since it is designed for HR data. The authors should consider more baseline models in the experiments.\n\n### Other concerns\nBesides, I am also concerned with the claims on knowledge, data and experimental settings:\n* The authors claimed that “Notably, this network is characterized with the capability to encode given physics knowledge”; “The overall network architecture mimics the forward Euler time stepping”; “The recurrent Π-block is a major innovation of this network architecture to capture the dynamics”. What is the difference between PeRCNN and PDE-net and PDE-net 2.0 (using the kernel to represent the derivative term, and use the δt-block to mimic the forward Euler time stepping)?\n* The author mentioned that “The obtained coefficients from sparse regression may not fully exploit all the available measurement as the regression is performed on subsampled data.”. Sparse regression is based on HR data, why is it subsampled? Why not use all HR data directly in the sparse regression, but perform additional fine-tuning? If the essence of fine-tuning is coefficient fitting, then the only influencing factor should be the amount of data.\n* The author mentioned that “the dominant reconstruction part lacks explicit physical interpretability (i.e., grey-box model), which may induce the difficulty in optimization especially for high-dimensional PDE systems with delicate patterns.”. Why does the process of generating HR data need to be interpretable? It seems that the PDE discovery can be realized as long as the process of mining PDE from HR data is interpretable.\n* The author should use the same candidate set in different PDE missions. It seems that the current candidate set is designed according to the PDE to be discovered. Besides, the number of candidates is relatively small, which reduces the difficulty of the problem. In practical applications, it is difficult to give a specific candidate set for a certain mission since we do not know the form of the PDE in advance. The candidate set should contain all possible candidates, and should not change according to the mission.\n\n### Minor points:\nThere are two “the” repeated at the end of the ninth line on the third page.\n\n### References\n1. Xu, H., Chang, H., & Zhang, D. (2020). DLGA-PDE: Discovery of PDEs with incomplete candidate library via combination of deep learning and genetic algorithm. Journal of Computational Physics, 418, 109584.\n2. Maslyaev, M., Hvatov, A., & Kalyuzhnaya, A. (2019, June). Data-driven partial derivative equations discovery with evolutionary approach. In International Conference on Computational Science (pp. 635-641). Springer, Cham.\n3. Udrescu, S. M., & Tegmark, M. (2020). AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16), eaay2631.\n4. Chen, Y., Luo, Y., Liu, Q., Xu, H., & Zhang, D. (2021). Any equation is a forest: Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE). arXiv preprint arXiv:2106.11927.\n5. Luo, Y., Liu, Q., Chen, Y., Hu, W., & Zhu, J. (2021). KO-PDE: Kernel Optimized Discovery of Partial Differential Equations with Varying Coefficients. arXiv preprint arXiv:2106.01078.\n6. Long, Z., Lu, Y., Ma, X., & Dong, B. (2018, July). Pde-net: Learning pdes from data. In International Conference on Mac", "summary_of_the_paper": "This paper proposes a deep convolutional-recurrent network plus a sparse regression model for discovering spatiotemporal PDEs. The three-dimensional PDE experimenta l results are used to validate the effectiveness.", "summary_of_the_review": "This paper addressed an important problem, PDE-discovery and was clearly written. Some points are new but there are some critical issues on novelty, experiments, knowledge and data which suggests a reject.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LMh2n-yhLXU", "reviewer": "Reviewer_53sM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- 1. Consider defining all terms in equation (1). Here, x, t, u_x, u_y. Also, notation such as \\nabla^2 should be defined in a separate section (in the Appendix) and cited early on (nabla is first defined in section 4.1). Also, the acronym I/BCs is not defined. \n\n- 2. The product block (PB) seems to be central to the discussion but this block is not motivated in the manuscript. Why did the authors choose this architecture? What is the role of PB?\n\t- a. While eq. (2) utilizes u_k to form \\mathcal{F}_hat(u_k), Fig. 1b shows mapping of u_k to u_k. This is pretty confusing. This can be mitigated by labeling 1b according to (2). In other words, where is (2) in Fig. 1b?\n\t- b. It seems that \\mathcal{F}_hat(.) is this supposed to mimic action of \\mathcal{F}(.), however it is unclear why it should take this form.\n\t- c. Essentially, it seems that Fig. 1b is trying to emulate the forward Euler time stepping, as opposed to this being the action of the overall network, since ISG generates HR from LR. If so, consider correcting para. 2 on page 4. \n\t- d. The terminology \"highway physics-based Conv\" layers needs citation. \n\n- 3. In Section 3.3:\n\t- a.  u is defined as a n_s x n_t matrix while in in equation (1) it is a vector of size n. This is pretty confusing since the discussion so far considers u as a vector. Consider updating the notation in (1) by stating the more general spatiotemporal problem. \n\t- b. Regarding the problem stated in (4) -- \n\t\t- i) Why do the authors solve (4) via STRidge, when they can use Iterative Hard Threasholding (IHT) (Haupt and Nowak (2006), Blumensath and Davies (2009)) directly? Arguably, there is no need to solve the l2-regularized least squares problem first, which may lead to faster solutions. This choice needs to be justified adequately.  \n         - ii) Also, (4) is the l0-regularized least squares problem. There is a rich body of literature devoted to l0-minimization, and needs citations to some iconic works, and potentially a section of related works devoted to it (in the Appendix).\n\t\t- iii) The paper claims that STRidge \"iteratively searches for the optimal tolerance based on the selection criteria (4)\". STRidge provides a sparse estimate for a given tolerance, and does not conduct a search. It may be that the authors conducted a search over various choices of the tolerances. If so, consider correcting this statement.\n\n- 4. In Section 3.4:\n\t- a. u_tilde is a 3d tensor of dimension here. It is unclear why this is a tensor. Consider clarifying and also, define H' and W'. \n\t- b. The discussion in \"Data reconstruction\" is very abrupt and does not tie-in with the discussion so far. This is exacerbated by the changing dimensions of u. How does this achieve interpolation? This weakens the entire discussion, so I highly recommend re-writing this part keeping the notation consistent. \n\t- c. What is an \"IC\" regularizer? \"IC\" is not defined.\n\t- d. What are u_0 and u_tilde?\n\t- e. Does the sparse regression use the reconstructed HR observations? If so, should this use the italicized capital U notation?\n\t- f. Last para. pg. 5 -- \"The obtained coefficients from sparse regression may not fully exploit all the available measurement as the regression is performed on subsampled data.\" -- In the discussion preceding this statement the sparse regression was performed on the HR observations. \n\t\t- i) Why and when was the data sub-sampled?\n\t\t- ii) Why do the authors initially form the HR data when the Sparse regression is performed on the subsampled data?\n\t\t- iii) Also, this may be partly due to the use of hard thresholding on the l2-regularized least squares. The l2-regularization may not let the coefficients to scale properly. \n\n\t- e. How are these components trained? Although individual components are described, there is no discussion on how these work together or the training procedure. I am assuming these components are trained separately? Consider adding a discussion about this.\n\t- f. The role of recurrent network in 1b is unclear. The various vertically arranged conv blocks result from the final u_k? Consider clarifying this. \n\n- 5. The authors claim that one of the advantages of the work is its interpretability. Is it due to the interpretability due to the sparse coefficients being able to identify the components of the pde? If yes, then:\n\t- a. there should be a discussion of the interpretability properties.\n\t- b. the experiments should demonstrate how the recovered coefficients and the  \n\n6. I recommend adding a baseline solving l0-regularized least squared directly using IHT by removing the l2 regularization. This is essential to analyze the performance of the proposed method. \n\nReferences:\n- HAUPT, J. and NOWAK, R. (2006). Signal reconstruction from noisy random projections. IEEE Transactions on Information Theory, 52 4036–4048\n- BLUMENSATH, T. and DAVIES, M. E. (2009). Iterative hard thresholding for compressed sensing.\nApplied and Computational Harmonic Analysis, 27 265–274", "summary_of_the_paper": "The paper focuses on discovering the underlying physics dynamics, in the form of partial differential equation (PDE) from low-resolution (LR) measurements. It leverages sparse regression-based recovery to identify the constituents of the unknown PDE. The main contribution is a module that constructs high-resolution (HR) observations from LR data, for sparse regression to perform well.", "summary_of_the_review": "Although the paper proposes these modules and tries to solve an interesting problem, it is unclear how these modules work together. Also, they use a sparse regression method (STRidge) which seems like a round-about way of solving the problem directly, which may potentially be faster. The paper is well-written in parts with some notational issues and incomplete description. That being said, I hope the comments are useful for updating the manuscript in the response period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a pipeline to discover PDEs from scarce and noisy data. Reviewers engaged in a very thoughtful discussion with the authors. I read the extensive rebuttal, and I believe the authors have addressed the major concerns claimed by the reviewers. I ask the authors to make sure to include all the changes and additional experiments in the camera-ready version.
Decision: Accept (Poster)