Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality

1 INTRODUCTION . Partial differential equations ( PDEs ) play a prominent role in many disciplines of science and engineering . The recent deep learning breakthrough and the rapid development of sensors , computational power , and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods ( Long et al. , 2018 ; 2019 ; Raissi et al. , 2019 ; Han et al. , 2018 ; Sirignano & Spiliopoulos , 2018 ; Khoo et al. , 2017 ) , especially in high dimensions where conventional methods become impractical . The set of applications that motivate this interest is wide-ranging , including computational physics ( Han et al. , 2018 ; Long et al. , 2018 ; Raissi et al. , 2019 ) , inverse problem ( Zhang et al. , 2018 ; Gilton et al. , 2019 ; Fan & Ying , 2020 ) and quantitative finance ( Heaton et al. , 2017 ; Germain et al. , 2021 ) . The numerical methods generated by the use of deep learning techniques are mesh-less methods , see the discussion in ( Xu , 2020 ) . A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage ( when available ) of a variational formulation , whose solution coincides with the solution of the PDE of interest . Despite the success and popularity of adopting neural networks for solving high-dimensional PDEs , the following question still remains poorly answered . For a given PDE and a data-driven approximation architecture , how large the sample size and how complex the model are needed to reach a prescribed performance level ? In this paper , we aim to establish the numerical analysis of such deep learning based PDE solvers . Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number , known as the neural scaling law ( Kaplan et al. , 2020 ; Hestness et al. , 2017 ; Sharma & Kaplan , 2020 ) , we aim to explore the neural scaling law for deep PDE solvers and compare its performance to Fourier approximation . Among the various approaches of using deep learning methods for solving PDEs , in this work , we focus on the Deep Ritz method ( DRM ) ( E & Yu , 2018 ; Khoo et al. , 2017 ) and the Physics-Informed Neural Networks ( PINN ) approach ( Sirignano & Spiliopoulos , 2018 ; Raissi et al. , 2019 ) , both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs . To provide theoretical guarantees for DRM and PINN , following ( Lu et al. , 2021b ; Duan et al. , 2021 ; Bai et al. , 2021 ) , we decompose the error into approximation error ( Yarotsky , 2017 ; Suzuki , 2018 ; Shen et al. , 2021 ) and generalization error ( Bartlett et al. , 2005 ; Xu & Zeevi , 2020 ; Farrell et al. , 2021 ; Schmidt-Hieber et al. , 2020 ; Suzuki , 2018 ) . However , instead of the O ( 1/ √ n ) ( n is the number of data sampled ) slow rate generalization bounds established in prior work ( Lu et al. , 2021b ; Shen et al. , 2021 ; Xu , 2020 ; Shin et al. , 2020 ) , we utilize the strongly convex structure of the DRM and PINN objectives and provide an O ( 1/n ) fast rate generalization bound ( Bartlett et al. , 2005 ; Xu & Zeevi , 2020 ) that leads us to a non-parametric estimation bound . Our theory also suggests an optimal selection of network size with respect to the number of sampled data . Moreover , to illustrate the optimality of our upper bound , we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modified version of DRM . We also test our theory by numerical experiments . Recent works ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Mikami et al. , 2021 ) studying a variety of deep learning algorithms all find the same polynomial scaling relation between the testing error and the number of data . As the number of training data n increases , the population loss L of well-trained and well-tuned models scales with n as a power-law L ∝ 1nα for some α . ( Sharma & Kaplan , 2020 ) also scans over a large range of α and problem dimension d and finds an approximately α ∝ 1d scaling law . In Section 4 , we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory . 1.1 RELATED WORKS . Neural Scaling Law The starting point of our work is the recent observation across speech , vision and text ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Rosenfeld , 2021 ) that the empirical performance of a model satisfies a power law scales as a power-law with model size and dataset size . ( Sharma & Kaplan , 2020 ) further finds out that the power of the scaling law depends on the intrinsic dimension of the dataset . Theoretical works ( Schmidt-Hieber et al. , 2020 ; Suzuki , 2018 ; Suzuki & Nitanda , 2019 ; Chen et al. , 2019b ; Imaizumi & Fukumizu , 2020 ; Farrell et al. , 2021 ; Jiao et al. , 2021c ) explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network . Our work extends this line of research to solving PDEs . Deep Network Based PDE Solver . Solving high dimensional partial differential equations ( PDEs ) has been a long-standing challenge due to the curse of dimensionality . At the same time , deep learning has shown superior flexibility and adaptivity in approximating high dimensional functions , which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing . Recent years , pioneer works ( Han et al. , 2018 ; Raissi et al. , 2019 ; Long et al. , 2018 ; Sirignano & Spiliopoulos , 2018 ; Khoo et al. , 2017 ) try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks ( Lu et al. , 2021a ; Li et al. , 2020 ) . Based on the natural idea of representing solutions of PDEs by ( deep ) neural networks , different loss functions for solving PDEs are proposed . ( Han et al. , 2018 ; 2020 ) utilize the Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network ( Zang et al. , 2020 ) solves the weak formulations of PDEs via an adversarial network . In this paper , we focus on the convergence rate of the Deep Ritz Method ( DRM ) ( E & Yu , 2018 ; Khoo et al. , 2017 ) and Physics-Informed neural network ( PINN ) ( Raissi et al. , 2019 ; Sirignano & Spiliopoulos , 2018 ) . DRM ( E & Yu , 2018 ; Khoo et al. , 2017 ) utilizes the variational structure of the PDE , which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs , and trains a neural network to minimize the variational objective . PINN ( Raissi et al. , 2019 ; Sirignano & Spiliopoulos , 2018 ) trains a neural network directly to minimize the residual of the PDE , i.e. , using the strong form of the PDE . Theoretical Guarantees For Machine Learning Based PDE Solvers . Theoretical convergence results for deep learning based PDE solvers raises wide interest recently . Specifically , ( Lu et al. , 2021b ; Grohs & Herrmann , 2020 ; Marwah et al. , 2021 ; Wojtowytsch et al. , 2020 ; Xu , 2020 ; Shin et al. , 2020 ; Bai et al. , 2021 ) investigate the regularity of PDEs approximated by neural network and ( Lu et al. , 2021b ; Luo & Yang , 2020 ) further provide a generalization analysis . ( Nickl et al. , 2020 ) introduces a prior over the solution of the PDE and considers an equivalent white noise model ( Brown & Low , 1996 ) . ( Nickl et al. , 2020 ) provides the rate of convergence of the posterior . Our paper does not need to introduce the prior on the target function and provides a non-asymptotic guarantee for finite number of data . At the same time , ( Nickl et al. , 2020 ) can only be applied to linear PDEs while our proof technique can be extended to nonlinear ones . All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate . ( Hütter & Rigollet , 2019 ; Manole et al. , 2021 ) consider the similar problem for the optimal transport problem , i.e . Monge-ampere equation . Nevertheless , the variational problem we considered is different from ( Hütter & Rigollet , 2019 ; Manole et al. , 2021 ) and leads to technical difference . The most related works to ours are two concurrent papers ( Duan et al. , 2021 ; Jiao et al. , 2021a ; b ) . However , our upper bound is faster than ( Duan et al. , 2021 ; Jiao et al. , 2021a ; b ) . In this paper , we also show that generalization analysis in ( Lu et al. , 2021b ; Duan et al. , 2021 ; Luo & Yang , 2020 ) are loose due to the lack of a localization technique ( De Boor & De Boor , 1978 ; Bartlett et al. , 2005 ; Koltchinskii , 2011 ; Xu , 2020 ) . With observation of the strong convexity of the loss function , we follow the fast rate results for ERM ( Schmidt-Hieber et al. , 2020 ; Xu & Zeevi , 2020 ; Farrell et al. , 2021 ) and provide a near optimal bound for both DRM and PINN . 1.2 CONTRIBUTION . In short , we summarize our contribution as follows • In this paper , we first considered the statistical limit of learning a PDE solution from sampled observations . The lower bound shows a non-standard exponent different from non-parametric estimation of a function . • Instead of the O ( 1/ √ n ) slow rate generalization bounds in ( Lu et al. , 2021b ; Duan et al. , 2021 ; Jiao et al. , 2021a ; c ) , we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods ( Van De Geer , 1987 ; Bartlett et al. , 2005 ; Koltchinskii , 2011 ; Srebro et al. , 2010 ; Xu & Zeevi , 2020 ) . We discovered that the current Deep Ritz Methods is sub-optimal and propose a modified version of it . We showed that PINN and the modified version of DRM can achieve nearly min-max optimal convergence rate . Our result is listed in Table 1 . • We tested the recently discovered neural scaling law ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Hashimoto , 2021 ) for deep PDE solvers numerically . The empirical results verified our theory .

Summary: This paper studies the the statistical error of the Deep Ritz Method and Physics-Informed Neural Networks using neural networks and truncated Fourier basis in solving PDEs. The static Schrodinger equation is used as a prototype PDE. With appropriate assumptions, the authors established upper and lower bounds of the error for both methods. The upper bound derived in this paper improves existing results with a faster rate. The authors also proved that the upper bound of PINN is nearly optimal. Some numerical experiments are conducted to verify the results.