Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: PINNs often fail to converge. The authors propose to use neural tangent kernel to solve this issue, and propose to use the Adam optimizer. They empirically demonstrate this to work on problems with high-frequency features.
strength_and_weaknesses: # Strengths

- for the first time, PINNs under SGDM, and Adam are analyzed, and their relation to solving spectral bias is discussed
- theoretical analysis

# Weaknesses

- it has been shown that the Adam optimizer works better than vanilla SGD in many cases (unrelated to PINNs), why would you assume that it does not work for PINNs?
- the paper aims to solve spectral bias. However, the phenomenon of spectral bias is not clearly defined or introduced in the paper
- figures are of bad quality



minor issues:
- spelling mistakes (week -> weak)
clarity,_quality,_novelty_and_reproducibility: # clarity

The main motivation of the paper, the existence of spectral bias, is not introduced very well. It is only mentioned that it is a known problem, but the referenced papers do not give clear definitions of spectral bias either. There should be some discussion of how to measure spectral bias, and what it is. Otherwise the paper is fairly well written, except for some typos and grammatical mistakes. The figures have bad quality and are too small.

# quality

seems to be good

# novelty

the analysis is important, but novelty is limited, there is no specific new method introduced. 

# reproducibility

code is not given, but should be possible to reproduce from the paper
summary_of_the_review: PINNs and their convergence behaviour is an understudied area where any insights can be very valuable to future research. The present work attempts to do some fundamental work to investigate PINNs. It is good to do some theoretical analysis, but novelty is limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper the authors study the convergence of infinitely wide PINNs on PDEs that have high frequency modes to examine the effects of spectral bias. Their analysis proceeds using NTK theory of convergence. They show that while SGD can learn high frequency modes that the learning rate has to be small. Further they show that SGD with momentum as well as ADAM reduces the effects of spectral bias and speed up the learning. The paper also features empirical results on the poisson equation, reaction-diffusion equation and transport function.  
strength_and_weaknesses: -The authors do a good job of introducing the reader to spectral bias in a general context in addition to spectral bias in the case of PINNs.---Very clear and concise analysis for section 3.1 and 3.2.  
-Nice experimental results, they seem to confirm theoretical convergence results. 


-The authors could flush out section 3.3; maybe move some of the material in the appendix for convergence results into main paper and list equations in format similar to 3.1 and 3.2
-Would be nice if authors concocted a numerically tractable ay to extract exact eigenvalues of Hessians to see how it matches with theory.

clarity,_quality,_novelty_and_reproducibility: I think the paper is clearly written and results are reproducible. I don't believe anyone has used NTK to analyze PINNs for this context. 
summary_of_the_review: Overall a good paper, easy to understand what the authors aimed to do. Seems like an important and useful result and ties in considering the spectrum of the physics problem in consideration with convergence guarantees of the network.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper studies the spectral bias phenomenon when using momentum-based methods to optimize physical-informed neural networks. The authors simplify the neural networks as a linear model (i.e., NTK), and prove that the continuous approximation of GDM converges to the high-frequency solution faster than SGD. The authors also investigate the convergence of Adam over PINN through a specific model. Experiment results are provided to show that Adam and SGDM converge faster than SGD when solving the PINNs.



strength_and_weaknesses: **Weakness**

1. The biggest weakness of this paper is that the analysis does not utilize the property of physical-informed loss. In other words, the same results also hold for image classification tasks as the analysis does not rely on which loss is used. This makes the result hard to unveil any information specific to the NN-solving-PDE task.

2. There are some overclaims in the paper. It is said that "using the SGD with momentum (SGDM) optimizer can reduce
the effect of spectral bias in the networks", but the paper only proves the result for a continuous approximation of GDM with no approximation error provided.

2. The paper is hard to follow, and many places lack explanations. For example, at the end of page 3, it is said "the norm of the Hessian $\mathcal{H}(w_t)$ is order $\mathcal{O}(C^4)$". However, the parameterization of $u(w,x)$ is not even provided (although it is explained in the appendix). On page 4, it is said that "the decay rate analysis becomes more involved as Eq. 7". However, there is no Eq. 7 in the main text. Also, why does using SGDM lead to faster convergence for high-frequency components than SGD? How do we know it from Theorem 1?
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Based on the evaluation above, I believe that this paper is below the bar of ICLR. The paper can be improved by refining the writing and the theoretical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper uses the NTK to discuss the effect of momentum on the training success of PINNs. It is shown that momentum (or adaptive momentum) can improve the convergence speed compared to standard SGD-based training, using both theoretical and empirical arguments.
strength_and_weaknesses: The paper is well-written and covers an interesting and timely topic. While the authors do not propose a novel method, their experimental evidence is in favor of their hypothesis. The supplementary material contains additional information that helps shedding light on the interplay between the spectral properties of the solution, the selection of the optimizer, and training success.

What prevents me from giving a better recommendation is that some of the experimental evidence is unconvincing, and that -- I believe -- alternative hypotheses first need to be ruled out. Indeed, judging from the results, it appears as if spectral bias is not the problem underlying convergence problems. Looking at, e.g., Fig. E.2 or Fig. 2, we see that SGD manages to learn the high-frequency components, but with a shift, trend, or offset added to it. Therefore, the spectral bias, which suggests that the high-frequency components are learned later, seems not to be a valid explanation. As an alternative hypothesis, it may be that learned solutions $u$ satisfy the PDE. While it is known that trivial (zero) solutions are attractive for PINNs (arXiv:2109.09338,arXiv:2203.13648), I assume that the same holds for non-trivial solutions that satisfy the PDE. Essentially, I argue that the solution $u$ provided by the PINN satisfies the PDE, but not the initial/boundary conditions. This needs to be confirmed or rejected by investigating the loss term $\mathcal{L}_r(w)$ throughout training, and not only the compound loss $\mathcal{L}$. If this former loss is small, then the PDE is at least approximately satisfied.

Further, the results in Fig. E.4 seem questionable, as a standard neural network should be capable of learning this simple function. Please correct me if I am missing something important.
clarity,_quality,_novelty_and_reproducibility: ### Quality, Novelty, and Reproducibility:
The paper treats an interesting topic and the perspective about the effects of momentum is definitely novel (to the best of my knowledge). However, some important questions regarding the experimental evidence and possible alternative hypotheses need to be taken care of to increase the paper's quality.

### Clarity:
Generally, the paper is clearly written. Below are pointers to a few exceptions, that can be taken care of easily.
- End of Sec. 2.1: What does "optimal" in "optimal solution $u(x,w)$ refer to? The solution of the PDE, or the minimum of the optimization loss landscape? These may be different.
- Equations at the beginning of Sec. 2.2: There seems to be something wrong. If we insert $u_{h-1}$ into $g_h$, then the weight matrix $\Theta_{h-1}$ is applied twice. Please check.
- In eq. (4), what are $s(x)$ and $b(x)$?
- In Th. 1, what is $m$? Can $m$ be chosen arbitrarily, or does it depend on the parameters of the PDE/the PINN?

### Minor Comments:
- "they are limited to some week empirical evidence" -> weak
- Proposition A -> Proposition 1
- After Th. 1, the references to eq. (7) should be a reference to eq. (6), right?
- In Sec. 4.3, "unperformed" -> underperformed; Section (4) -> Section 4; the results for the Poisson equation with $C=10\pi$ are described twice.

*EDIT* Increased score during discussion period.
summary_of_the_review: An interesting paper that studies the effect of momentum on the training success of PINNs. While interesting and novel, the experimental evidence is not yet fully convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper received two borderline scores, and two other, very pronounced scores. Of the latter two, the positive review mentions three specific strengths of the paper:  
(i) a good introduction to spectral (both in a general context and specifically for PINNs)
(ii) clear and concise analysis for the theory sections 3.1 and 3.2.
(iii) convincing experimental results, which seem to confirm the theory

On the other hand, the very negative review mentions three weak points:
(iv) the theory part is not specifically related to PINNs, since the results depend only on the eigenvalues of kernel matrix K. Therefore, the specific relevance of the theoretical derivation to PINNs is not obvious.
(v)  the role and interpretation of the continuous approximation of GDM is unclear, due to the dependency of a decaying learning rate and  increasing rate of parameter alpha.
(vi) some parts of the paper are hard to follow. Among other points of criticism, it was not fully clear how and why momentum could accelerate the training. As an example, the under-damped case with a convergence rate of GDM of order  exp(-gamma t) was mentioned, where the rate of GD is essentially the same.

After the rebuttal and discussion phase, I had the general impression that both the very positive and the very negative score were a bit too extreme. Therefore, I tried to largely ignore these numerical values and to focus only on the depth of the positive and negative arguments.    My conclusion is that the first positive point (i) was shared by all reviewers, whereas the second on third one were more controversial.  Concerning (iii), one of the "borderline" reviewers initially said that "some of the experimental evidence is unconvincing", although this impression seems to have changes slightly after the rebuttal. This improvement on the experimental side, however, did not change the over-all impression of this paper: the reviewer was still not overly positive in the end. 

Concerning point (ii), there was quite some disagreement, which brings us to the discussion of the critical points (iv) to (vi). I think, that all three points of criticism are somewhat valid, maybe to a different degree. Concerning the over-all conceptual idea,  I think that particularly (iv) and (vi) are severe weaknesses that could not be properly addressed in the rebuttal. In my opinion, the authors' argument that the the results were still highly specific to PINNs although the theory does not differentiate between the two types of eigenvalues ("collocation" vs  "IC/BC") is neither clear nor very convincing. 
The importance of point (v) is less clear to me, and might to some degree be explained by the fact that different (sub-)communities have different notions of what "is common in the field". But I would argue that in such a situation, a more in-depth discussion of this issue would be needed in the paper. 
Concerning the last point (vi) about the exponential convergence rates, I had the impression that the reviewer and the authors simply talk about different things, namely the convergence to the stationary solution expressed by the real part of the eigenvalues and the transient solution, i.e. the oscillatory behavior described by the imaginary parts, and neither the reviewer nor myself were fully satisfied with the authors' rebuttal in this regard.     

After finally going over all reviews and discussions again, I think that the critical points are somewhat substantial, and that there are many open questions remaining after the rebuttal. Further, also one of the "neutral" reviewers finally indicated that he/she was "not fully convinced by the correctness of the authors' hypothesis", and even the most positive reviewer did not want to champion this paper for acceptance. Therefore, I vote for rejection.
justification_for_why_not_higher_score: There are certainly several interesting aspects in this paper, but also several open questions, and to me, these open questions were so substantial that I voted for rejection.
Decision: Reject