Discovering Nonlinear PDEs from Scarce Data with Physics-encoded Learning

1 INTRODUCTION . Deriving physical laws remains critical for understanding the dynamical patterns and complex spatiotemporal behaviors in nature . Well formulated governing equations facilitate scientific discovery and promote establishment of new disciplines . Traditionally , scientists seek to find governing equations by rigorously following the first principles , including the conservation laws , geometric brownian motion assumptions and knowledge-based deductions . However , these classical approaches show limited capability and slow progress in identifying clear analytical governing equations , and leave numerous complex systems under-explored . Thanks to the recent achievements in machine learning ( ML ) , advances in governing equation/law discovery has been drastically accelerated . The pioneer works ( Bongard & Lipson , 2007 ; Schmidt & Lipson , 2009 ) apply symbolic regression to reveal the underlying differential equations that govern nonlinear dynamical systems without any prior background knowledge . Although this inspiring investigation implies the dawn of discovering the fundamental principles automatically in scientific ML , the scalability and overfitting issues herein have been the critical concerns . Recently , a ground-breaking work by ( Brunton et al. , 2016 ) introduces the sparse regression into the discovery of parsimonious governing equations ( aka. , Sparse Identification of Nonlinear Dynamics ( SINDy ) ) , based on the assumption that the dynamical systems are essentially controlled by only few dominant terms , which evolves to promote the sparsity in the candidate functions . Furthermore , the introduction of spatial derivative terms and the improvement of the sparsity-promoting algorithm as Sequential Threshold Ridge regression ( STRidge ) make SINDy applicable to general PDE discovery ( i.e. , the PDE-FIND algorithm ) ( Rudy et al. , 2017 ) . However , due to the demanding requirement of derivative estimation via numerical differentiation , these sparse regression methods largely rely on high-quality measurement data , which is barely accessible in the real-world sensing environment . Therefore , there is an urgent need to address the issue of discovering the underlying governing equations from low-resolution ( LR ) and noisy measurement data . Very recently , many works attempt to relieve the pain by reconstructing the high-resolution ( HR ) scientific data from LR measurements and then discovering the governing laws , including robust low-rank discovery ( Li et al. , 2020a ) , physics-informed spline learning ( Sun et al. , 2021 ) and physics-informed deep learning ( Chen et al. , 2021 ) . Nevertheless , the dominant reconstruction part lacks explicit physical interpretability ( i.e. , grey-box model ) , which may induce the difficulty in optimization especially for high-dimensional PDE systems with delicate patterns . Contributions . To overcome the challenges resulted from the noisy and LR measurements , we propose a novel physics-encoded deep learning framework ( i.e. , white-box model ) for discovering the governing equations of physical systems . The main contributions of this paper are three-fold : • We design a novel recurrent network architecture that is able to fit spatiotemporal measurement data for HR metadata generation with high accuracy and physical consistency . Notably , this network is characterized with capability to encode given physics knowledge ( e.g. , known terms , assumed PDE structure based on multiplicative feature operation , initial/boundary conditions ) so that the given physics as model prior is respected rigorously . • Based on the proposed reconstruction network , we develop a hybrid optimization approach to discover spatiotemporal PDEs from noisy and LR measurements . Specifically , to determine the explicit form of underlying governing equations , sparse regression is performed by using HR prediction inferred from the model . Finally , through inheriting the PDE terms and coefficients from the sparse discovery , we conduct the fine-tuning step with the completely physics-based network to further improve the accuracy of the scientific discovery . • The proposed coupled framework establishes the physics-encoded data-driven predictive model from the noisy and LR measurement . To evaluate the performance of the model , we perform three numerical experiments on a variety of nonlinear systems . It is found that the proposed approach demonstrates excellent robustness and accuracy in spite of the high level of noise and the scarcity ( spatially and temporally ) in the measurements . 2 RELATED WORK . Dynamical system modeling . Simulating multi-dimensional PDE systems with neural networks ( NNs ) has been a renewed research topic , which can be dated back to last century ( Lee & Kang , 1990 ; Lagaris et al. , 1998 ) . Like the traditional simulation methods , the recent breakthrough in NNbased modeling for physical systems also fall into two streams : continuous and discrete learning . The continuous learning scheme can avoid the notorious demand of time-step and grid sizes for numerical stability . Among the meshfree strategies , based on the incorporation of prior knowledge of PDE systems , there exist two directions seeking for the approximation of simulation with DNNs , i.e. , the pure data-driven approaches ( Han et al. , 2018 ; Long et al. , 2018 ; Wang et al. , 2020 ) and physics-informed learning ( Yu et al. , 2017 ; Raissi et al. , 2019 ; Rao et al. , 2021b ; Karniadakis et al. , 2021 ) . The former option relies on large amounts of high-quality data , while the physics-informed ML only requires scarce or even no labeled data due to the enhancement from physical constraints . Besides , it is worthwhile to mention that the recent studies on neural operators ( Li et al. , 2020b ; Lu et al. , 2021 ; Patel et al. , 2021 ) have exhibited the great potential of learning the nonlinear , meshfree and infinite-dimensional mapping with DNNs for physical systems . Regarding the discrete learning scheme , the mesh-based methods play a dominant role in simulating complex high-dimensional PDE systems . Herein , the convolutional architectures are mainly applied for handling regular grids in steady-state problems ( Zhu & Zabaras , 2018 ; Zhu et al. , 2019 ) and spatiotemporal systems ( Bar-Sinai et al. , 2019 ; Geneva & Zabaras , 2020 ; Kochkov et al. , 2021 ; Ren et al. , 2021 ) , as well as irregular domains by considering elliptic coordinate mapping ( Gao et al. , 2021a ) . Moreover , with the theoretical development of graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) , researchers seek for a new direction for geometry-adaptative learning of nonlinear PDEs with arbitrary domains ( Belbute-Peres et al. , 2020 ; Sanchez-Gonzalez et al. , 2020a ; Gao et al. , 2021c ) . In addition to the common mesh-based approaches , the particle-based frameworks have been attracting a lot of interest in scientific modeling for physical systems , especially by employing GNNs ( Li et al. , 2018 ; Ummenhofer et al. , 2019 ; Sanchez-Gonzalez et al. , 2020b ) . Scientific data augmentation . Scientific data are typically sparse , noisy and incomplete . Therefore , data augmentation has been an active research area and received massive attention in the scientific ML community , e.g. , data reconstruction from sparse measurements ( Raissi , 2018 ; Callaham et al. , 2019 ; Erichson et al. , 2020 ; Rao et al. , 2021a ) , super-resolution ( SR ) ( Stengel et al. , 2020 ; Gao et al. , 2021b ) and denoising ( Fathi et al. , 2018 ) . Generally , based on the type of incorporated data , we categorize the existing data augmentation methods into two groups . The first one is to extract the coherent structure and correlation features as the basis by utilizing the sufficient high- quality datasets . For instance , in the context of SR , the majority of NN-based frameworks ( Liu et al. , 2020 ; Esmaeilzadeh et al. , 2020 ; Fukami et al. , 2021 ) aim to reconstruct the high-resolution ( HR ) full-field data from low-resolution ( LR ) measurements by employing HR ground truth reference as the constraint . Nevertheless , the second type of data augmentation takes advantage of the physicsbased model and can accurately capture the high-fidelity dynamical patterns from the LR and noisy measurement data . The recent advances of physics-based learning models have shown the the great success of data augmentation for PDE systems with only sparse and noisy data ( Raissi , 2018 ; Rao et al. , 2021a ) and even without any labeled data ( i.e. , only with PDEs ) ( Gao et al. , 2021b ) . Data-driven discovery . The data-driven discovery is also a renascent research attempt . The earliest work ( Dzeroski & Todorovski , 1995 ) introduced the inductive logic programming to find the natural laws from experimental data . With the rapid advance of machine intelligence , the past three decades have seen a considerable amount of literatures growing up around the interplay between ML and scientific discovery . Essentially , two genres have been observed in distilling governing equations , with respect to the searching of candidate functions . Firstly , the natural and optimal solution to automatically identify the governing equations for dynamical systems is to learn a symbolic model from experimental data . Hence , symbolic regression ( Schmidt & Lipson , 2009 ) and symbolic neural networks ( SNN ) ( Sahoo et al. , 2018 ; Kim et al. , 2020 ) have been explored for inferring the physical models without any prior knowledge . In addition , GNN ( Cranmer et al. , 2020 ) has also been studied to identify the nontrivial relation due to its excellent inductive capability . The second group manually defines a large library functions based on the prior knowledge , and utilizes the sparse-prompting algorithms to discover the parsimonious physics model efficiently . The most representative works are SINDy ( Brunton et al. , 2016 ) and PDE-FIND ( Rudy et al. , 2017 ) for ordinary differential equations ( ODEs ) and PDEs , respectively . Nevertheless , the standard sparse representation-based methods are limited to the high-fidelity noiseless measurement , which is usually difficult and expensive to obtain . Hence , many recent efforts have been devoted to discover PDEs from sparse/noisy data ( Li et al. , 2020a ; Xu et al. , 2021 ; Sun et al. , 2021 ; Chen et al. , 2021 ) . 3 METHODOLOGY . 3.1 PROBLEM DESCRIPTION . Discovering the explicit form of the governing equation ( s ) for a given system from the measurement data is of critical importance for scientists to understand some underexplored processes . To formalize this problem , which is also known as the data-driven equation discovery , let us assume the response of a multi-dimensional spatiotemporal system is described by ut = F ( x , t , u , u , ux , uy , ∇u , · · · ) ( 1 ) where x = [ x , y ] T and t denote the spatial and temporal coordinates respectively , u denotes the state variable possibly consisting of multiple components ( e.g. , u has two components u and v ) and the subscript denotes the partial derivatives , e.g. , ux = ∂u/∂x . F is a function parameterizing the system dynamics with possible linear/nonlinear terms ( e.g. , u2 , ux and ∇u , where the Nabla operator ∇ is defined as [ ∂/∂x , ∂/∂y ] T ) . The objective of the data-driven equation discovery is to uncover the closed form of F provided time series measurements on some fixed spatial locations . Although this problem has been studied extensively over the past few years , the combinatorically large search space of the possible equation appears as one of the major obstacles that hinders the discovery of the governing equation from very limited measurements . The existence of noise , as well as the coarse-sampling of the measurement , also adds to the difficulty of this problem . To address these challenges , we proposed a novel physics-encoded recurrent network that is able to reconstruct the high-fidelity solution from the noisy and LR measurements with the help of prior physics knowledge . We successfully combine this network with the sparse regression for discovering the explicit form of the governing PDE of dynamical systems .

Summary: The focus of this paper is to develop a learning framework to discover spatiotemporal PDEs from scarce and noisy data.  The authors suggest devise a deep convolutional-recurrent network which encodes prior physics knowledge followed by  sparse regression with reconstructed data to identify the analytical form of the governing PDEs. The framework is validated  on three high-dimensional PDE systems where superiority over baselines is demonstrated.