Discovering Nonlinear PDEs from Scarce Data with Physics-encoded Learning

1 INTRODUCTION . Deriving physical laws remains critical for understanding the dynamical patterns and complex spatiotemporal behaviors in nature . Well formulated governing equations facilitate scientific discovery and promote establishment of new disciplines . Traditionally , scientists seek to find governing equations by rigorously following the first principles , including the conservation laws , geometric brownian motion assumptions and knowledge-based deductions . However , these classical approaches show limited capability and slow progress in identifying clear analytical governing equations , and leave numerous complex systems under-explored . Thanks to the recent achievements in machine learning ( ML ) , advances in governing equation/law discovery has been drastically accelerated . The pioneer works ( Bongard & Lipson , 2007 ; Schmidt & Lipson , 2009 ) apply symbolic regression to reveal the underlying differential equations that govern nonlinear dynamical systems without any prior background knowledge . Although this inspiring investigation implies the dawn of discovering the fundamental principles automatically in scientific ML , the scalability and overfitting issues herein have been the critical concerns . Recently , a ground-breaking work by ( Brunton et al. , 2016 ) introduces the sparse regression into the discovery of parsimonious governing equations ( aka. , Sparse Identification of Nonlinear Dynamics ( SINDy ) ) , based on the assumption that the dynamical systems are essentially controlled by only few dominant terms , which evolves to promote the sparsity in the candidate functions . Furthermore , the introduction of spatial derivative terms and the improvement of the sparsity-promoting algorithm as Sequential Threshold Ridge regression ( STRidge ) make SINDy applicable to general PDE discovery ( i.e. , the PDE-FIND algorithm ) ( Rudy et al. , 2017 ) . However , due to the demanding requirement of derivative estimation via numerical differentiation , these sparse regression methods largely rely on high-quality measurement data , which is barely accessible in the real-world sensing environment . Therefore , there is an urgent need to address the issue of discovering the underlying governing equations from low-resolution ( LR ) and noisy measurement data . Very recently , many works attempt to relieve the pain by reconstructing the high-resolution ( HR ) scientific data from LR measurements and then discovering the governing laws , including robust low-rank discovery ( Li et al. , 2020a ) , physics-informed spline learning ( Sun et al. , 2021 ) and physics-informed deep learning ( Chen et al. , 2021 ) . Nevertheless , the dominant reconstruction part lacks explicit physical interpretability ( i.e. , grey-box model ) , which may induce the difficulty in optimization especially for high-dimensional PDE systems with delicate patterns . Contributions . To overcome the challenges resulted from the noisy and LR measurements , we propose a novel physics-encoded deep learning framework ( i.e. , white-box model ) for discovering the governing equations of physical systems . The main contributions of this paper are three-fold : • We design a novel recurrent network architecture that is able to fit spatiotemporal measurement data for HR metadata generation with high accuracy and physical consistency . Notably , this network is characterized with capability to encode given physics knowledge ( e.g. , known terms , assumed PDE structure based on multiplicative feature operation , initial/boundary conditions ) so that the given physics as model prior is respected rigorously . • Based on the proposed reconstruction network , we develop a hybrid optimization approach to discover spatiotemporal PDEs from noisy and LR measurements . Specifically , to determine the explicit form of underlying governing equations , sparse regression is performed by using HR prediction inferred from the model . Finally , through inheriting the PDE terms and coefficients from the sparse discovery , we conduct the fine-tuning step with the completely physics-based network to further improve the accuracy of the scientific discovery . • The proposed coupled framework establishes the physics-encoded data-driven predictive model from the noisy and LR measurement . To evaluate the performance of the model , we perform three numerical experiments on a variety of nonlinear systems . It is found that the proposed approach demonstrates excellent robustness and accuracy in spite of the high level of noise and the scarcity ( spatially and temporally ) in the measurements . 2 RELATED WORK . Dynamical system modeling . Simulating multi-dimensional PDE systems with neural networks ( NNs ) has been a renewed research topic , which can be dated back to last century ( Lee & Kang , 1990 ; Lagaris et al. , 1998 ) . Like the traditional simulation methods , the recent breakthrough in NNbased modeling for physical systems also fall into two streams : continuous and discrete learning . The continuous learning scheme can avoid the notorious demand of time-step and grid sizes for numerical stability . Among the meshfree strategies , based on the incorporation of prior knowledge of PDE systems , there exist two directions seeking for the approximation of simulation with DNNs , i.e. , the pure data-driven approaches ( Han et al. , 2018 ; Long et al. , 2018 ; Wang et al. , 2020 ) and physics-informed learning ( Yu et al. , 2017 ; Raissi et al. , 2019 ; Rao et al. , 2021b ; Karniadakis et al. , 2021 ) . The former option relies on large amounts of high-quality data , while the physics-informed ML only requires scarce or even no labeled data due to the enhancement from physical constraints . Besides , it is worthwhile to mention that the recent studies on neural operators ( Li et al. , 2020b ; Lu et al. , 2021 ; Patel et al. , 2021 ) have exhibited the great potential of learning the nonlinear , meshfree and infinite-dimensional mapping with DNNs for physical systems . Regarding the discrete learning scheme , the mesh-based methods play a dominant role in simulating complex high-dimensional PDE systems . Herein , the convolutional architectures are mainly applied for handling regular grids in steady-state problems ( Zhu & Zabaras , 2018 ; Zhu et al. , 2019 ) and spatiotemporal systems ( Bar-Sinai et al. , 2019 ; Geneva & Zabaras , 2020 ; Kochkov et al. , 2021 ; Ren et al. , 2021 ) , as well as irregular domains by considering elliptic coordinate mapping ( Gao et al. , 2021a ) . Moreover , with the theoretical development of graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) , researchers seek for a new direction for geometry-adaptative learning of nonlinear PDEs with arbitrary domains ( Belbute-Peres et al. , 2020 ; Sanchez-Gonzalez et al. , 2020a ; Gao et al. , 2021c ) . In addition to the common mesh-based approaches , the particle-based frameworks have been attracting a lot of interest in scientific modeling for physical systems , especially by employing GNNs ( Li et al. , 2018 ; Ummenhofer et al. , 2019 ; Sanchez-Gonzalez et al. , 2020b ) . Scientific data augmentation . Scientific data are typically sparse , noisy and incomplete . Therefore , data augmentation has been an active research area and received massive attention in the scientific ML community , e.g. , data reconstruction from sparse measurements ( Raissi , 2018 ; Callaham et al. , 2019 ; Erichson et al. , 2020 ; Rao et al. , 2021a ) , super-resolution ( SR ) ( Stengel et al. , 2020 ; Gao et al. , 2021b ) and denoising ( Fathi et al. , 2018 ) . Generally , based on the type of incorporated data , we categorize the existing data augmentation methods into two groups . The first one is to extract the coherent structure and correlation features as the basis by utilizing the sufficient high- quality datasets . For instance , in the context of SR , the majority of NN-based frameworks ( Liu et al. , 2020 ; Esmaeilzadeh et al. , 2020 ; Fukami et al. , 2021 ) aim to reconstruct the high-resolution ( HR ) full-field data from low-resolution ( LR ) measurements by employing HR ground truth reference as the constraint . Nevertheless , the second type of data augmentation takes advantage of the physicsbased model and can accurately capture the high-fidelity dynamical patterns from the LR and noisy measurement data . The recent advances of physics-based learning models have shown the the great success of data augmentation for PDE systems with only sparse and noisy data ( Raissi , 2018 ; Rao et al. , 2021a ) and even without any labeled data ( i.e. , only with PDEs ) ( Gao et al. , 2021b ) . Data-driven discovery . The data-driven discovery is also a renascent research attempt . The earliest work ( Dzeroski & Todorovski , 1995 ) introduced the inductive logic programming to find the natural laws from experimental data . With the rapid advance of machine intelligence , the past three decades have seen a considerable amount of literatures growing up around the interplay between ML and scientific discovery . Essentially , two genres have been observed in distilling governing equations , with respect to the searching of candidate functions . Firstly , the natural and optimal solution to automatically identify the governing equations for dynamical systems is to learn a symbolic model from experimental data . Hence , symbolic regression ( Schmidt & Lipson , 2009 ) and symbolic neural networks ( SNN ) ( Sahoo et al. , 2018 ; Kim et al. , 2020 ) have been explored for inferring the physical models without any prior knowledge . In addition , GNN ( Cranmer et al. , 2020 ) has also been studied to identify the nontrivial relation due to its excellent inductive capability . The second group manually defines a large library functions based on the prior knowledge , and utilizes the sparse-prompting algorithms to discover the parsimonious physics model efficiently . The most representative works are SINDy ( Brunton et al. , 2016 ) and PDE-FIND ( Rudy et al. , 2017 ) for ordinary differential equations ( ODEs ) and PDEs , respectively . Nevertheless , the standard sparse representation-based methods are limited to the high-fidelity noiseless measurement , which is usually difficult and expensive to obtain . Hence , many recent efforts have been devoted to discover PDEs from sparse/noisy data ( Li et al. , 2020a ; Xu et al. , 2021 ; Sun et al. , 2021 ; Chen et al. , 2021 ) . 3 METHODOLOGY . 3.1 PROBLEM DESCRIPTION . Discovering the explicit form of the governing equation ( s ) for a given system from the measurement data is of critical importance for scientists to understand some underexplored processes . To formalize this problem , which is also known as the data-driven equation discovery , let us assume the response of a multi-dimensional spatiotemporal system is described by ut = F ( x , t , u , u , ux , uy , ∇u , · · · ) ( 1 ) where x = [ x , y ] T and t denote the spatial and temporal coordinates respectively , u denotes the state variable possibly consisting of multiple components ( e.g. , u has two components u and v ) and the subscript denotes the partial derivatives , e.g. , ux = ∂u/∂x . F is a function parameterizing the system dynamics with possible linear/nonlinear terms ( e.g. , u2 , ux and ∇u , where the Nabla operator ∇ is defined as [ ∂/∂x , ∂/∂y ] T ) . The objective of the data-driven equation discovery is to uncover the closed form of F provided time series measurements on some fixed spatial locations . Although this problem has been studied extensively over the past few years , the combinatorically large search space of the possible equation appears as one of the major obstacles that hinders the discovery of the governing equation from very limited measurements . The existence of noise , as well as the coarse-sampling of the measurement , also adds to the difficulty of this problem . To address these challenges , we proposed a novel physics-encoded recurrent network that is able to reconstruct the high-fidelity solution from the noisy and LR measurements with the help of prior physics knowledge . We successfully combine this network with the sparse regression for discovering the explicit form of the governing PDE of dynamical systems .

Summary: Problem: data-driven PDE discovery methods are not robus to low-quality measurement data.  Solution: A novel architecture that can encode prior knowledge (known terms, PDE structure, boundary conditions), and sparse regression procedure that is hypothesized to be more robust to scarce and noisy data scenarios.  Other Contributions: * proof that proposed Pi-block is a universal polynomial approximator. This is why I scored a "4" on technical contributions.