Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality

1 INTRODUCTION . Partial differential equations ( PDEs ) play a prominent role in many disciplines of science and engineering . The recent deep learning breakthrough and the rapid development of sensors , computational power , and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods ( Long et al. , 2018 ; 2019 ; Raissi et al. , 2019 ; Han et al. , 2018 ; Sirignano & Spiliopoulos , 2018 ; Khoo et al. , 2017 ) , especially in high dimensions where conventional methods become impractical . The set of applications that motivate this interest is wide-ranging , including computational physics ( Han et al. , 2018 ; Long et al. , 2018 ; Raissi et al. , 2019 ) , inverse problem ( Zhang et al. , 2018 ; Gilton et al. , 2019 ; Fan & Ying , 2020 ) and quantitative finance ( Heaton et al. , 2017 ; Germain et al. , 2021 ) . The numerical methods generated by the use of deep learning techniques are mesh-less methods , see the discussion in ( Xu , 2020 ) . A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage ( when available ) of a variational formulation , whose solution coincides with the solution of the PDE of interest . Despite the success and popularity of adopting neural networks for solving high-dimensional PDEs , the following question still remains poorly answered . For a given PDE and a data-driven approximation architecture , how large the sample size and how complex the model are needed to reach a prescribed performance level ? In this paper , we aim to establish the numerical analysis of such deep learning based PDE solvers . Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number , known as the neural scaling law ( Kaplan et al. , 2020 ; Hestness et al. , 2017 ; Sharma & Kaplan , 2020 ) , we aim to explore the neural scaling law for deep PDE solvers and compare its performance to Fourier approximation . Among the various approaches of using deep learning methods for solving PDEs , in this work , we focus on the Deep Ritz method ( DRM ) ( E & Yu , 2018 ; Khoo et al. , 2017 ) and the Physics-Informed Neural Networks ( PINN ) approach ( Sirignano & Spiliopoulos , 2018 ; Raissi et al. , 2019 ) , both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs . To provide theoretical guarantees for DRM and PINN , following ( Lu et al. , 2021b ; Duan et al. , 2021 ; Bai et al. , 2021 ) , we decompose the error into approximation error ( Yarotsky , 2017 ; Suzuki , 2018 ; Shen et al. , 2021 ) and generalization error ( Bartlett et al. , 2005 ; Xu & Zeevi , 2020 ; Farrell et al. , 2021 ; Schmidt-Hieber et al. , 2020 ; Suzuki , 2018 ) . However , instead of the O ( 1/ √ n ) ( n is the number of data sampled ) slow rate generalization bounds established in prior work ( Lu et al. , 2021b ; Shen et al. , 2021 ; Xu , 2020 ; Shin et al. , 2020 ) , we utilize the strongly convex structure of the DRM and PINN objectives and provide an O ( 1/n ) fast rate generalization bound ( Bartlett et al. , 2005 ; Xu & Zeevi , 2020 ) that leads us to a non-parametric estimation bound . Our theory also suggests an optimal selection of network size with respect to the number of sampled data . Moreover , to illustrate the optimality of our upper bound , we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modified version of DRM . We also test our theory by numerical experiments . Recent works ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Mikami et al. , 2021 ) studying a variety of deep learning algorithms all find the same polynomial scaling relation between the testing error and the number of data . As the number of training data n increases , the population loss L of well-trained and well-tuned models scales with n as a power-law L ∝ 1nα for some α . ( Sharma & Kaplan , 2020 ) also scans over a large range of α and problem dimension d and finds an approximately α ∝ 1d scaling law . In Section 4 , we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory . 1.1 RELATED WORKS . Neural Scaling Law The starting point of our work is the recent observation across speech , vision and text ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Rosenfeld , 2021 ) that the empirical performance of a model satisfies a power law scales as a power-law with model size and dataset size . ( Sharma & Kaplan , 2020 ) further finds out that the power of the scaling law depends on the intrinsic dimension of the dataset . Theoretical works ( Schmidt-Hieber et al. , 2020 ; Suzuki , 2018 ; Suzuki & Nitanda , 2019 ; Chen et al. , 2019b ; Imaizumi & Fukumizu , 2020 ; Farrell et al. , 2021 ; Jiao et al. , 2021c ) explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network . Our work extends this line of research to solving PDEs . Deep Network Based PDE Solver . Solving high dimensional partial differential equations ( PDEs ) has been a long-standing challenge due to the curse of dimensionality . At the same time , deep learning has shown superior flexibility and adaptivity in approximating high dimensional functions , which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing . Recent years , pioneer works ( Han et al. , 2018 ; Raissi et al. , 2019 ; Long et al. , 2018 ; Sirignano & Spiliopoulos , 2018 ; Khoo et al. , 2017 ) try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks ( Lu et al. , 2021a ; Li et al. , 2020 ) . Based on the natural idea of representing solutions of PDEs by ( deep ) neural networks , different loss functions for solving PDEs are proposed . ( Han et al. , 2018 ; 2020 ) utilize the Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network ( Zang et al. , 2020 ) solves the weak formulations of PDEs via an adversarial network . In this paper , we focus on the convergence rate of the Deep Ritz Method ( DRM ) ( E & Yu , 2018 ; Khoo et al. , 2017 ) and Physics-Informed neural network ( PINN ) ( Raissi et al. , 2019 ; Sirignano & Spiliopoulos , 2018 ) . DRM ( E & Yu , 2018 ; Khoo et al. , 2017 ) utilizes the variational structure of the PDE , which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs , and trains a neural network to minimize the variational objective . PINN ( Raissi et al. , 2019 ; Sirignano & Spiliopoulos , 2018 ) trains a neural network directly to minimize the residual of the PDE , i.e. , using the strong form of the PDE . Theoretical Guarantees For Machine Learning Based PDE Solvers . Theoretical convergence results for deep learning based PDE solvers raises wide interest recently . Specifically , ( Lu et al. , 2021b ; Grohs & Herrmann , 2020 ; Marwah et al. , 2021 ; Wojtowytsch et al. , 2020 ; Xu , 2020 ; Shin et al. , 2020 ; Bai et al. , 2021 ) investigate the regularity of PDEs approximated by neural network and ( Lu et al. , 2021b ; Luo & Yang , 2020 ) further provide a generalization analysis . ( Nickl et al. , 2020 ) introduces a prior over the solution of the PDE and considers an equivalent white noise model ( Brown & Low , 1996 ) . ( Nickl et al. , 2020 ) provides the rate of convergence of the posterior . Our paper does not need to introduce the prior on the target function and provides a non-asymptotic guarantee for finite number of data . At the same time , ( Nickl et al. , 2020 ) can only be applied to linear PDEs while our proof technique can be extended to nonlinear ones . All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate . ( Hütter & Rigollet , 2019 ; Manole et al. , 2021 ) consider the similar problem for the optimal transport problem , i.e . Monge-ampere equation . Nevertheless , the variational problem we considered is different from ( Hütter & Rigollet , 2019 ; Manole et al. , 2021 ) and leads to technical difference . The most related works to ours are two concurrent papers ( Duan et al. , 2021 ; Jiao et al. , 2021a ; b ) . However , our upper bound is faster than ( Duan et al. , 2021 ; Jiao et al. , 2021a ; b ) . In this paper , we also show that generalization analysis in ( Lu et al. , 2021b ; Duan et al. , 2021 ; Luo & Yang , 2020 ) are loose due to the lack of a localization technique ( De Boor & De Boor , 1978 ; Bartlett et al. , 2005 ; Koltchinskii , 2011 ; Xu , 2020 ) . With observation of the strong convexity of the loss function , we follow the fast rate results for ERM ( Schmidt-Hieber et al. , 2020 ; Xu & Zeevi , 2020 ; Farrell et al. , 2021 ) and provide a near optimal bound for both DRM and PINN . 1.2 CONTRIBUTION . In short , we summarize our contribution as follows • In this paper , we first considered the statistical limit of learning a PDE solution from sampled observations . The lower bound shows a non-standard exponent different from non-parametric estimation of a function . • Instead of the O ( 1/ √ n ) slow rate generalization bounds in ( Lu et al. , 2021b ; Duan et al. , 2021 ; Jiao et al. , 2021a ; c ) , we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods ( Van De Geer , 1987 ; Bartlett et al. , 2005 ; Koltchinskii , 2011 ; Srebro et al. , 2010 ; Xu & Zeevi , 2020 ) . We discovered that the current Deep Ritz Methods is sub-optimal and propose a modified version of it . We showed that PINN and the modified version of DRM can achieve nearly min-max optimal convergence rate . Our result is listed in Table 1 . • We tested the recently discovered neural scaling law ( Hestness et al. , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2019 ; Hashimoto , 2021 ) for deep PDE solvers numerically . The empirical results verified our theory .

Summary: This paper establishes statistical lower bounds and upper bounds for (a modified) Deep Ritz method and PINNs based learning of solutions of PDEs when the estimators belong either to a class of sparse neural networks or lie in truncated fourier basis. They utilize the fact that the objective in DRM and PINNS is strongly convex, and use it to get a faster generalization bound O(1/n) instead of O(1/\sqrt{n}). Given that the upper bound for the initial non-modified version of DRM does not match the lower bound, they introduced a modified deep ritz method, where the number of samples to estimate the gradient squared is greater than (the ration is provided in the statement) rest of the objective. This enables them to achieve minimax optimality for DRM as well.   Through their experiments they verify that the number of training samples n and the test error follows a power low with \alpha = 1/d as indicated by the derived rates.