Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper establishes statistical lower bounds and upper bounds for (a modified) Deep Ritz method and PINNs based learning of solutions of PDEs when the estimators belong either to a class of sparse neural networks or lie in truncated fourier basis. They utilize the fact that the objective in DRM and PINNS is strongly convex, and use it to get a faster generalization bound O(1/n) instead of O(1/\sqrt{n}). Given that the upper bound for the initial non-modified version of DRM does not match the lower bound, they introduced a modified deep ritz method, where the number of samples to estimate the gradient squared is greater than (the ration is provided in the statement) rest of the objective. This enables them to achieve minimax optimality for DRM as well. 

Through their experiments they verify that the number of training samples n and the test error follows a power low with \alpha = 1/d as indicated by the derived rates.
main_review: The paper is well motivated and understanding the generalization and optimality rates for NNs for PDEs is an important problem. Moreover, the authors provide a lower bound which is important. 

However, I would like to point out that currently the way the paper is presented is very hard to follow. Most of the discussion and proof sketches are left for the appendix, and the theorems that are presented in the main paper are informally stated. Furthermore, there is not proof sketch provided with the theorems that would intuit how the result was proved. This makes it hard to verify the results. 

For example, like the previous work (Duan et al., 2022 and Jiao et al., 2021) the authors use a spline based construction to prove the neural network approximability. However the authors point out that they get better bounds due to the strong convexity of DRM and PINN loss. However, I am not sure where exactly is it helping when compared to the proof sketch provided in the previous paper? An explanation of this in the main paper is lacking. 

Regarding the lower bound, I am quite confused about what authors mean by "The lower bound shows a non-standard exponent different from non-parametric estimation of a function." 
summary_of_the_review: I think the results of the paper are relevant however, I think that the exposition needs to be improved for the main paper, as currently it is hard to follow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Applying deep learning (DL) to solving PDEs numerically has been a very exciting research directions. The current paper studies certain statistics properties of approximating linear Elliptic PDEs using neural networks (and truncated Fourier series). Under certain (quite strong) assumptions on the function class, the authors proved sharper (in some cases, tight)  bounds for the approximation errors. 
main_review: The paper studies approximation power of DL-based PDE solvers, mainly the Deep Ritz Methods (DRM) and Physics-informed neural neural network method (PINN). A key observations is proposition 2.1/2.2, in which generalization/approximation error is upper/lower bounded tightly by certain energy functional $ E = E(u, |\nabla u|, \Delta u, ...)$. As such, the problems themself are reduced to the classical machine learning setting:  **generalization bound = complexity bound + approximation bound**. The paper proceeds (in the appendix) to control these two using local Rademacher complexity argument (where assumptions of uniform bounded are needed), B-splines, covering numbers... (The appendix is not very well organized!)

A couple comments of the experimental sections. 

(1) Sec 6.3 seems not directly related to the paper (PDE solver). They are quite universally observed in DL. 
(2) Could you provide a link for the code/ colab? 

Crucial assumptions for Theorem 4.2, 4.2 etc have been mentioned in the main text, though it is brought up in discussion section. 

[Strength] The paper provides several bounds (rigorous proof) for the approximation error for deep-learning-based PDE solvers, some of these bounds are tight. 


[Weakness] 
(1) The assumptions (e.g. eq (A.43), the functions, derivatives are UNIFORMLY bounded) are quite strong as acknowledged by the authors, which rule out many interesting cases and remove higher order terms in the proof.  
(2) The paper is about approximation / representation power of neural networks. Optimization and thus optimization-related-generalization are not covered by the paper. 




Minor: 
(1) The appendix is long. It is not easy to find the proof of each theorem in the appendix. Please provide a pointer to the proof of each theorem. 
(2) Given the length of the appendix ~ 50 pages, it should have a content page, a brief introduction to walk the readers through its main structure, what is the main theme of each section, etc.  
summary_of_the_review: DL-based PDE solvers are important and exciting direction of deep learning. The current paper provides several insights about the statistical properties of DL-based solver. Although there are several limitation (mentioned above), I think the contribution is significant enough for a ICLR publication and many researchers in the ICLR community will find it insightful (at least for me)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper carries out a variety of studies, both theoretical and numerical, on numerical solution of a Schrodinger equation using deep learning inspired methods.  The main results are upper and lower bounds on a power law scaling for sample complexity, function of dimension and regularity, which are tight for one of the methods.  Another method (Deep Ritz) has a proposed improvement.
main_review: A number of recent works (many but not all cited by the authors) analyze ML-inspired numerical methods for PDE, with one
outstanding question being to what extent these methods evade the curse of dimension.  The present work uses far more sophisticated techniques from statistical learning theory, in particular a "localization technique for Rademacher complexity sums", and gets stronger results.  While not leading to dramatically new conclusions, these are apparently the first tight bounds for these problems.
summary_of_the_review: Valuable technical advances on an important and popular topic.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the the statistical error of the Deep Ritz Method and Physics-Informed Neural Networks using neural networks and truncated Fourier basis in solving PDEs. The static Schrodinger equation is used as a prototype PDE. With appropriate assumptions, the authors established upper and lower bounds of the error for both methods. The upper bound derived in this paper improves existing results with a faster rate. The authors also proved that the upper bound of PINN is nearly optimal. Some numerical experiments are conducted to verify the results.
main_review: Strength: 
1. The authors derived faster upper bound for PINN and DRM than existing results.

2. The authors proposed a modified DRM. The modified method together with truncated Fourier basis method.

3. The proof is very detailed and clear.


Weakness:
1. Compared with the network architecture in (Jiao et. al., 2021a) and (Duan et. al., 2021), the architecture used in this paper is a sparse network, which makes it less practical. 

2. Another draw back is that the authors assumes the gradient of any function in the network class is bounded by a constant. This assumption is difficult to satisfy since the gradient depends on the network width and depth. In the theorems, the depth depends on the number of samples.

3. The organization of the paper needs to be improved. I believe the main theorems are Theorem A10, A11, A13 and A14. However, the authors only stated informal versions of these theorems in the body part. Even in these theorem mentioned above, the conditions are unclear. For example, in Theorem A9, the authors require the network class has bounded gradient. But this assumption is not mentioned in A10 and A11, which I believe is proved based on A9. The assumption in Proposition 2.1 is not mentioned in A10 and A11, either.

4. For numerical experiments, the authors only presented the errors versus the number of training data. The network architecture and how it is trained are not mentioned. What is the computational cost?

Comments:
1. On page 3, the authors said the variational problem considered in this paper and that considered in (Hutter & Rigollet, 2019; Manole et al, 2021) are different and leads to technical difference. Could the authors comment on the difference?

2. On page 3, the authors said the proof in this paper can be extended to nonlinear ones. Could the authors briefly explain how to extend it?

3. Page 15: At the end of the proof of Theorem A.1, the right-hand side of the last inequality has a factor max(1,V_{min}). The inequality does not hold when V<1. But I think this can be fixed.

Typo:
page 3: '... can be extend to ...'-> '... can be extended to ...'
summary_of_the_review: In general, this paper is a good paper and provides solid theoretical results on solving PDEs by deep neural networks. But the writing and organization need to be improved.
review_title: null

Meta-review: This paper proposes a new theory for modified DRM and PINN for solving elliptical PDEs, and delivers valuable advances on important topics.
Decision: Accept (Poster)