Consistency of Physics-Informed Neural Networks for Second-Order Elliptic Equations
Authors: Yuqian Cheng
Zhuo Chen
Qian Lin
Venue: NeurIPS 2025 poster ()
Keywords: consistency
PINN
kernel gradient flow
second-order elliptic equation
Primary area: deep_learning

Abstract: The physics-informed neural networks (PINNs) are widely applied in solving differential equations. However, few studies have discussed their consistency. In this paper, we consider the consistency of PINNs when applied to second-order elliptic equations with Dirichlet boundary conditions. We first provide the necessary and sufficient condition for the consistency of the physics-informed kernel gradient flow algorithm, and then as a direct corollary, when the neural network is sufficiently wide, we obtain a necessary and sufficient condition for the consistency of PINNs based on the neural tangent kernel theory. We also estimate the non-asymptotic loss bounds of physics-informed kernel gradient flow and PINN under suitable stronger assumptions. Finally, these results inspires us to construct a notable pathological example where the PINN method is inconsistent.

Consistency of Physics-Informed Neural Networks for Second-Order Elliptic Equations
Yugian Cheng Department of Mathematical Science, Tsinghua University cyq21 @mails. ts tsinghua .edu. cn
Zhuo Chen Department of Mathematical Science, Tsinghua University chenzhuo@ts inghua. edu. cn
Qian Lin Department of Statistics and Data Science,
Tsinghua University qianl ilannnnn@ttn edu cn
Abstract
The physics-informed neural networks (PINNs) are widely applied in solving differential equations. However, few studies have discussed their consistency. In this paper, consider the consistency of PINNs when applied second- order elliptic equations with Dirichlet boundary conditions. We first provide the necessary and sufficient condition for the consistency of the physics-informed kernel gradient flow algorithm. And then, as direct corollary, when the neural network is sufficiently wide, derive necessary and sufficient condition for the consistency of PINNs based on the neural tangent kernel theory. Additionally, we provide non-asymptotic loss bounds for physics-informed kernel gradient flow and PINN under suitable stronger assumptions. Finally, these results inspire construct a notable pathological example in which the PINN method is inconsistent.
1 Introduction
The basic settings of data-driven scientific computing problem usually start with a PDE problem in the following form:
Tu = in Ω Bu = on 82,
(1)
where Ω C Rd is the domain of the problem, T a differential operator, summarizes the boundary conditions, and the functions / and are the non-homogeneous terms of the problem. Assume that
(I) has a unique solution u..
Physics-infommnn informed machine learning [33] searches for numerical estimators of the solution to (I) by training a network ûg. called a phisoee-i neural network (PINN). Ideally, the network the trained to minimize the population PINN loss function, which is defined as the mean square residual of T
BBū (a) gxx) + / (T) (
L(ūa)=
00
However, this loss function 2 is numerically intractable, as all we can utilize are only finitely many X; 02, = N..) and {(Y,, + æÿ) Y, E Ω j = 1....,N; N, where 7/ and Ey independent noises. Thus, in practice, we train the network to minimize the following empirical version of PINN loss function:
samples {(X,.g(X) gXx) + m7)
N..
L(ūB)
(X, ((x)) 1212
Tūo (Yz) f(Y)) e,12.
(3)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Compared to traditional deep learning methods, physics- informe- neural networks (PINNs) feature unique training process that guided by physical laws described by partial differential equations. This distinctive approach is expected to combine the impressive approximation power and flexibility of neural networks with the robustness and intepretability of physics-informed modeling. Consequently, a thorough theoretical analysis of PINNs is crucial to provide mathematical guarantees for the efficiency of the corresponding algorithms and to offer guidance for model improvement.
In this paper, we discuss the consistency of PINNs, that is, the convergence of the population PINN loss to zero under appropriate conditions. We focus primarily on the convergence of PINNS for an important PDE problem: the second-order elliptic equation with homogeneous Dirichlet boundary condition. This special form of (T) where we take T be second-order elliptic operator C. Bu = u and g = O. To be more specific, we are mainly interested in the following question:
Question:
For the second-order elliptic equation with Dirichlet boundary condition, under what condition does the PINN usg converge to the PDE solution u".
In the study of the convergence of PINNs, challenges arise when analyze the training dynamics of PINNs, as the optimization of network training is typically a highly non-convex problem. A recent line of works such as [I [2] 30 34 focused on the generalization ability of some sort of empirical minimizer of the (regularized or non-regularized) PINN loss function. Compared with these prior works, in order to put the training dynamics of PINNs into consideration, in this paper introduce the framework of neural tangent kemel (NTK) ((21)) which connects the network training with kernel-based algorithm [ 99. In this framework, network trained by gradient flow converges to kernel gradient flow estimator of the kernel gradient flow model makes the neural tangent kernel powerful tool for analyzing training dynamics of wide networks. For example, [25] 277 proved the minimax optimality of wide multi-layer ReLU networks on bounded subdomains of Euclidean spaces.
28 32
as the width of network goes to infinity. The convexity
There exist some previous works that applied NTK theory to analyze PINNs, For example, 40
compute the training gradient flow of infinitely wide PINNs in the form of neural tangent kernel in order to analyze the spectral bias of PINNs. [5] applied the NTK theory and proved a convergence result for the empirical loss of wide two-layer PINNs along the training gradient flow. All of these prior works shed us some light on understanding the dynamics of the training process and analyzing the convergence of PINNs with great network width.
1.1 Our contributions
Our main theorem (Theorem 3.7 states the necessary and sufficient condition for the consistency of physics-informer kernel gradient flow. In this paper, the term "ccnsistency" refers to the convergence of the population PINN loss L to zero, and a formal definition of consistency is given in Definition kernel gradient flow is consistent to the problem (4) if
3.1 We will show that the and only if the solution u lies in an abstract space (defined in Definition 3.6). Then, using the technique of NTK (Theorem 5.4) we promote the main theorem to networks, and prove the sufficient and necessary conditions for the consistency of PINN (Theorem 5.8).
We list our technical contributions as follows, which provide essential details and key components of our main results:
(1) Although the main theorem is concerned with the convergence of PINN loss (2), we are still interested in the performance PINNs in terms of mean square loss. As will be shown in Lemma 3.9 we prove that the PINN loss can control the mean square loss.
(2) We present novel mathematical techniques to solve the ill-posed convex optimization problems, which the key to the estimation for the NTK gradient flow in our main results. To the best of our knowledge, this is the first convergence result for ill- ill-posed convex optimization on Hilbert space. See Lemma 4.2 and its remark for details.
(3) As application, we prove that, in the case of Poisson equations on sphere and torus, H. contains the Sobolev space H, (2) n H22),, which provides a convenient criterion for verifying whether the PINN method is consistent in this case. We note that this convergence result does not require the source-capacity assumption framework which is commonly adopted in prior works.
(4) As an important corollary of the main theorem, in Section 5.2 we construct a pathological example in which the PINN method is inconsistent.

---

containing convergence result of the population loss L (consistency)
involving analysis of network training
containing convergence result of the empirical loss L
involving NTK
dynamics
Yes
Yes (Theorem
This paper
Yes
Yes
(Lemmaa
4.1
and 4.2
3.7 and 5.8)
11
30 34
No
No
No
Yes
17
40
Yes
Yes
No
No
15 [18
26 42
Yes
Yes
Yes
No
Table 1: A table comparison between this work and related prior works mentioned above or below.
Remark 1.1. (About the terminology "consistency"" In this paper, we choose to adopt this term from the learning theory instead of directly using the more common term "convergence" mainly The term "consistency" emphasizes more on the convergence
due to the following reasons: of the population loss L, and we hope to distinguish the main result (Theorem 3.7p in this work from the convergence results of the empirical loss L 3 this work from the existing and forthcoming results that focus on the convergence rate, because the term "consisteney". in general, emphasizes conditions, while the convergence rates are stronger results under stronger conditions.
18 26 42 2. We hope distinguish
general results about convergence under weak
Background
2.1 Elliptic equations of second order
Our work focuses on the performance of PINNs on the problem of second-order elliptic equation with homogeneous Dirichlet boundary condition:
Cu(y) f(y), y E Z2 u(z) = 0, x E a.,
where f2 is a bounded subdomain of R f E (2( the nonhomogeneous term, and L is a second- order elliptic operator defined in the following divergence form: Lu(y) = Σnj= 0 (1)(()))(()) any (v) (ye(y). We assume the strong elliptic condition L. In other words, there is positive for any & Rd [ and Ω.
constant À > 0 such that Ei,s Diy (v)&
Let (22 be the closure of C (92) in the Sobolev space H' (2). For any f E (Q)), the problem (4) has a unique weak solution u² E (22) (see Section 8.2 of (19).
Denote a2 the boundary of Ω and Ω the closure of Ω. Assume that M2 is smooth, and ag E (Ω).. Then by the regularity theorem (Section 6.3 of (L4), for any f LI))),, the weak solution in the Sobolev space ( Moreover, the solution u* also strong solution (Chapter 9 of (I9).. In other words, u E C())), u* on 0n and Lu = =f a.e. !n.
2.2 Physics- Informed Neural Networks
We train a neural network úg to estimate the ground-truth solution w", The network ûg is set in the following structure:
(x) W: + b9. 0
E ERmii by "RR" p= 1=1,2;
[x) 1 (2 (x)) by W R U/ E Rm 1 1 L, p 1,2;
2p
/ml
(x(x) =
22 (2t (x)
I+ (x)).
11+1=1
(5)
When L = 1, üg is actually fully-connected network with one hidden layer with special initializa- tion 25). We also note that the NNK and NTK of ûi coincide with those fully-connected network (see Appendix F.3 for details). We initialize the parameters 0 (W! b3)) of úe in the following form
3

---

of mirror initialization, which is suggested in 8 25 27|
(W)),,= = (W))s, ~ 1),, (b))
1),,)),
t= 1= L.
(6)
This special initialization is adopted in this paper for only purpose: to ensure that úg 0 at the beginning of training. In fact, the choice of initialization is a delicate matter that needs careful attention in NTK theory, because it is proved in [8] that infinitely wide fully-connected network will not lie in the RKHS of its NTK at initial time, if all parameters are initialized as i.i.d. standard Gaussian random variables, in which case the NTK kernel gradient flow will no longer be mathematically well-defined. Thus, we adopted the special initialization (6) to avoid unnecessary disgression.
We also assume that the activation function sufficiently smooth such that Lüg is well-defined. Under the setting of (4). the PINN loss function L defined in 2 is now written in the following form:
/
|u(x)] de + J, [u(ly) [w)
L(u)=
on
(Here we set both the total measures of as and 12 be 1 after necessary rescaling.) The corresponding empirical loss function is
1 N, L(u)= = No
1 N
Σ [xx))+++
N Σ -u(Y))-f(Y)) -yy?:
(8)
where X; and Yy are i..d. sample points drawn from the uniform distributions of 0n and Q respectively, and e, are independent noises. (Here we set the noise T) on the boundary in (3) to be zero.)
Since the dimension of ö is smaller than the dimension of Ω we further assume that N₂ N₁.
2.3 Gradient Flow of Network Training and Neural Tangent Kernel
During training, the network parameter evolves along the following gradient flow:
Lo(t) - -V
_
(9)
Equivalently, the gradient flow of the function úg is given by
d
d
(o(e) (x) = (V) (t)
dt
dt
No.
N, Koog(aa (Cūo (CC - Ez),
Kote X,) Ea()) (X;)
Σ N₁
2
No
5=1
(10)
where Ko(a, v) = (Voūa<))) Vvūo(y) is called the neural network kemnel (NNK) function.
Here, the notation Ly Koce) (Iz, y) that means L is operated only the variable y. The notation C. is defined in the similar way, and clearly Koce) v) KetcyKeeya v)-
We will show in Section 5 that the NNK function Ko(t) (I, y) converges to the NTK function K KNT(2e (z. y) along the PINN gradient flow 11).. Moreover, the derivatives of Kęg) also converges to the derivatives of KNT respectively. Thus, it reasonable consider the following limit gradient flow:
N₁ Σ (N,N((H (L ((0,)))())))))))))))))))))))))>) ez).
d dt
2 N
2
(x)
KNT X3) (X.))
N₁
(11)
2.4 Reproducing Kernel Hilbert Space
Suppose that we have a positive-definite kernel function K(x, y) In other words, the integration
operator T : (2))) + L2(2), Tf(x) -fa K(x, (y)d has eigenvalues X > X d 2

---

0. Let o; be the corresponding cigenfunction of the eigenvalue A₃. By Mercer decomposition (38). ((x,g) = Li=o (z (v) converges absolutely and uniformly.
Denote K2(2)
((,, y), and define an inner product (,, the space {Ky y (2) by (Ky, y2 / (yh, (2). The closure of {Kn) under this inner product is denoted as H. called the reproducing kernel Hilbert space (RKHS) of K. This space is characterized as
H [(x) 1(x== - SEP
(12)
i=
and the inner product can be characterized by (f.g)H Ei fig, for f = Ein T, and g Ei=1 A, ()). Thus, the Hilbert space has natural isomorphism 12
2.5 Physics-Infrrmer Kernel Gradient Flow
With the language of RKHS, the limit gradient flow 11 can be interpreted in the following sense: Suppose that we aim to search for an estimator of the ground-truth solution u² in the RKHS of a kernel (xx,, v). Give u(x)= = 0o u. (z) E H. we will prove later that this expansion can be differentiated term by term: Lu(x)) Ei th La Consider H as parametric model space with parameters {- In other words, we parametrize an estimator û, E H by (x( = Σi= (((t)))) Then the gradient flow of the parameters along the time [0, 00) is d (t) 0u:
i=0 ah (ū.). and equivalently, the gradient flow of the function ún is
d
úx(x)= = Σ
20 Dü, d
(t(t) =-
80 Oûr ai
(ū.)
dt
Tu. dt
vū,
0ū;
(13)
N
2
K(x, X.) oz
N Σ (, (Ca, -(0)) -g))
2
N
We call this method the physics-informed kernel gradient flow. If we set the kernel function K(x; (x: y) to be the NTK function K KNT(I: (I, y). then (13) is precisely the gradient flow [1].. Thus, the training
of an infinitely wide PINN can be interpreted as the physics-informed informed kernel gradient flow of NTK.
3 Main Results
Before we state main results, need define the consistency formally:
Definition 3.1. (1) We say that the physics-informed kemel gradient flow (13) is consistent to the
problem 4 if lim,,, -oo L(ūr) = for a suitable training stopping time T = T(N,,N depending No and Ny.
(2) We say that the PINN method (described in Subsection 2.2
consistent to the problem 4 if Nim [(ūe(T) = 0 for suitable training stopping time T = T(Nu, (NNu,N Ny) and suitable network width m, =
I,..., L depending on N,, and Ny.
In other words, consistency means that we can select appropriate hyperparameters (network widths and stopping time) such that the PINN loss can be controlled to be arbitrarily small, provided that the sample size is sufficiently great. Remark 3.2. The PINN loss function L defines a natural metric
{
u1 u22 [ [uu - Luzl?
dpinn (u1, u2)
(14)
1022
called the PINN distance, and [uu) = dpinn(uu (u. u²)², hence is reasonable that we use the PINN loss I((a(t)) in Definition 3.1 to measure the distance between the PDE solution and the solution
estimator. As will be shown Lemma 3.9 the PINN distance can control the 2 distance.
We list the assumptions required follows:
Assumption 3.3. The kernel function K K(x, y) is pesitive-defintt.
5

---

Assumption 3.4. (x, y) is smooth enough so that 0g (x, y) is well-defined for any multi- indices a, 8 with |ax, [8] S 2.
Assumption 3.5. Assume that the term f in (4) is measurable and bounded, hence is also 1,2. Definition 3.6. Let H be the RKHS of K (ar, v), and let H be the closure of H in the Sobolev space (22 under the metric defined by (14).
We will show in Theorem A.1 that under Assumption 3.3 and Assumption 5.2 we have H. C C hence H well-defined.
3.1 Sufficient and necessary conditions of consistency
Now we state our main theorem:
Theorem 3.7. (Consistency of physics-informed kernel gradient flow) Let u² be the unique solution to 4., and let úe be the kernel gradient flow estimator defined in (13) Assume that the samples ( [N,) and (Yı.... YN,3 } are drawn from the uniform distributions of on and Q respectively, and Nu S 5. We further assume that Ej are independent wub-Gaussian noises.
Suppose that Assumption 3.3
3.4 and 3.5 are all satisfied. Let H the space defined in Definition
3.6 Then we huve:
If E H. then by setting the training stopping time to be T O(N.) for a E (0, 3), we have lim
L(ūT) + 0
(15)
No0
in probability, hence by Definition 3.2 the physics-informed kernel gradient flow is consistent in this
case;
(2) If & H. then
inf L(ū,) C>> 0 t.X;
(16)
for some positive constant C depending on Qt L, f and K. Here, the infimum is taken over all I [0, oo) and all possible choices of samples X;, Yj-
Remark 3.8. A trivial consequence of 16) that, if u² H. then for any possible stopping time T= T(Nu, N2), we have
lim inf L(ūr) C 0;
(17)
Nu
In other words, the physics-informed kernel gradient flow is always inconsistent in this case. Thus, Theorem 3.7 indicates that u" H is the necessary and sufficient condition for the consistency of physics-informed kernel gradient flow.
3.2 The Convergence of Mean Square Loss
It is natural to ask how close (o())) and the solution u' are in terms of mean square loss. Fortunately have the following 2 norm estimation:
Lemma 3.9. (PINN distance controls 1,2 distance) For any u E C²()),, we have
L(u) 2 Cllu - u² 1|222,
(18)
where the constant C' > 0 depends only on and
Proof. The proof is based on the technique of Green's function. See Appendix Dor details.
Remark 3.10. This estimation for 2 loss is nontrivial, and the standard elliptic estimation for weak solutions is far from sufficient obtain Lemma 3.9 because the kernel gradient flow estimator å or the PINN estimator uo(s) does not satisfy the Dirichlet boundary condition generally.
Since will shown in Theorem A.1 that H C then immediately obtain
Corollary 3.11. (L2 convergence) Under the same assumptions of Theorem 3.7
if PINN is consistent
44 L.e. H. then we also have
lim lim ||ur - u (122(00) + in probability. N Ng -oo m-+00
(19)

---

3.3 Non-Asymptotic Bound
Under the settings of our Theorem 3.7 the condition for consistency we provide is necessary and sufficient, which means that we characterize all of the cases where L(ūe) converges to O, including those very bad cases. In other words, the convergence speed can be arbitrarily low, hence it
impossible to obtain a non-asymptotic bound under the same settings of our main theorem.
However, if suitable stronger assumptions hold, then a non-asymptotic bound is accessible, as is shown in the following theorem:
Theorem 3.12. Under the same settings of Theorem 3.7 if we further assume that u' H. then with probability at least - -O(N"), we have
L(ūr) = -0(N. -- Vlog No + +Nu").
1 - u" (0le00)))>(0) ( ON
1-3e
Vlog No + NN Nu
(20)
at stopping time ((N.), (0. }). Here, the convention O hides all the terms involving K, Ω C and J.
Proof. See Appendix E
Example 3.13. A typical example of RKHS is the Sobolev space HTt for Sobolev kernel of (Q)) with will satisfy the condition u² E H.
13
Thus,
4 Proof of the Main Theorem
In contrast with traditional methods such as kernel ridge regression ( and kernel gradient flow where estimators be explicitly computed and directly analyzed, the peculiar form of the PINN loss function complicates the analysis of the gradient flow of network training. Many previous
44
works on kernel-based physics-ifformed modeling such [2 10
31 ignored the boundary term for convenience, while our work put the boundary condition into consideration. We estimate the gradient flow of PINN training in an indirect taste by studying the evolution of PINN loss function and approximating its derivative with respect to training time.
Our proof of Theorem (3.7 is based the decomposition L(ú,) = (L(0,) L(v))) + L(x),, where it is defined as the kernel gradient flow of the population loss 7
d Ji v.(()) = 2 / K(x, ve) - J. C, C, (Cv, - f(n))dn
(21)
The following two lemmata provide estimations for L(ū+) L(v,) and L(ve), respectively; Lemma 4.1. With probability ONN, 1.t.++N777 +N we have
(L(.,) - L(v)) (vv)l= = - log N₂ N
log No,
(22)
N₂
for all t E [0, Here, the randomness comes from the selection of samples.
Lemma 4.2. Along the gradient flow 227). we have L(vi) + inffen L(v) as lim; L(xx) = if and only if u² H.
Thus,
The proof's of the two lemmata are delayed to Appendix B and respectively.
Remark 4.3. Although Lemma 4.2 looks quite natural at first glance. the convergence of L(v:) is in fact highly nontrivial. This is because we do not assume that the minimizer u² lies in the model space H. but rather in larger space H. which makes the optimization problem ill-posed. Traditional results of convex optimization (6)) for example) are usually based on the sell-posedness; In other words, they assume that there exists minimizer of the target convex function the interior of the model domain. Lemma 4.2 shows that the PINN loss still converges to its infimum, even though there no minimizer in the RKHS and the RKHS norm i diverges to infinity along the gradient flow (2T). To the best of our knowledge, this is the first convergence result for specific ill-posed convex optimization problem on infiniteddimensonal Hilbert space. fact, the convergence of general ill-posed convex optimization on Hilbert space remains an open problem.
7

---

Now we are ready to prove the main theorem (Theorem 3.7p
Proof. (of the main theorem) u² E H. we take the stopping time to be T ( )
(0, 3). By Lemma 4.1 lim (L(ür) [(wr))) = 0 in probability; And by Lemma 4.2
N,,,..N N 420 (UT)) lim [(x, = 0 if and only if u² E H. Combining the three results above,
we obtain (15).
If u² $ H. since ūg E H for any 2 0 and any possible choices of samples X,, Yy. then L(ae) >
inf L(u) > O. Combining this estimation with Lemma 5.6 we obtain (16).
Results on PINN
As was discussed in Section 2 the technique of NTK would help promote results about kernel methods wide neural networks. In this section, we first prove the convergence NTK (Theorem
5.4) based on which we extend our main theorem (Theorem 3.7 7) to PINNs (see Theorem 5.8
The following assumptions are the analogues of Assumption 3.3 and 3.4
with respect to NTK:
Assumption 5.1. Assume that the NTK function KNT (described in 2.3 is positive-definit-
Assumption 5.2. Assume that E (R). Moreover, the derivatives of a are polynomially bounded: (r() C(1 Irl"), 0, 1,3, for some university constants C > 0 and p 1.
Remark 5.3. Many of the previous works have proposed criteria to verify the positiveness of neural tangent kernel K (TT y), such as [7]. The smoothness of is fundamental assumption to ensure that the derivatives of the network (e(C)) are well-defined. The polynomial bound condition is easily satisfied by various types of activation functions, such as tanh and ReLU for l: > 3.
Theorem 5.4. (Uniform convergence of NTK) Assume that Assumption 5.1 5.2 are satisfied.
(I) For a shallow network with only one hidden layer (L = 1 in [45 ), as the network width my + co, the NNK function Kate (Ir,y) y) converges in probability to the NTK function K K(x,, y) uniformly with respect x, y Q2 and t [] In other words.
sup telo,T),
KKoty (z, 3) KNT(x, - 0 in probability
(23)
as mum++ co. Here, the randomness arises from the initialization of network.
Moreover, LoKot) (z, (), (o v). Loc, also converge in probability to KNT (z, v), C NT (x, 3), C,C,KNN respect to t E [0, x. Ω.
respectively, and the convergence is uniform with
(II) Furthermore, i is conver, then the above results also holds for deep neural networks with depth Z 9 and activation function a tanh, the network widths satisfies em -m,.... m., S Cm for some 0 < c < C and m
oc.
Proof. The proof is delayed Appendix F
An experimental illustration for Theorem 5.4 (I) is provided in Figure
Remark 5.5. Theorem 5.4 (I) is the uniform version of Theorem 4.4 of 40 We expect that Theorem (II) remains valid for L = 2,3,..., while we defer the detailed proof to future work due to its technical complexity in computations. We offer numerical results that roughly verify Theorem 5.4 (II) for L = 4 in Appendix[]
Note that both (10) and (II) finite-order linear ODE systems. With the help of Theorem 5.4 and ODE theory, we show that the distance between the network úo(e)) and its corresponding physics- informed kernel gradient flow â, converges in probability 0:
Lemma 5.6. Under the same assumptions of Theorem 5.4 given No. N₂ and > O, as m + oo. have
sup sup (x)(0(() - xx(x)) + 0, = [6[0, X,Y,
sup sup (x) - Lúr(x) + (24) E te X,YY

---

in probability. Here, the randomness arises from the initialization of network.
Proof. The proof delayed Appendix
Remark 5.7. Both (2) and (x) depend the samples X,, Y,, and the notation supx,,y in
Lemma e.pphasiss the fact that the convergence is uniform with respect to all possible choices of samples X,, X, with sample sizes Ny fixed.
The experimental illustrations of this lemma can be found in Figure and 3 A quantitative version of this lemma for shallow networks
1) can be found in (330) the appendix.
As a direct corollary of Lemma 5.6 and the main theorem (Theorem (.7), we have the following theorem, which states the necessary and sufficient condition for the consistency of PINN:
Theorem 5.8. (Consistency of PINN) Let u' be the unique solution to (4). and let uo(s) be the
neural network the form (45). Assume that the samples (X1..... X N., and (Yi,...Y are drawn from the uniform distriuutions of 0h and Q respectively, and No. S &Ng.
Let HNTK be the RKHS of KNT, and let HYTK be the closure of HNTK under the metric dpsnn 14h. Then PINN is consistent to the problem (4) if and only if uf" E HNTK.
5.1 A Positive Example
Unfortunately, the computation of eigenfunctions of general NTK is still an open problem, hence Theorem[5.8js not applicable to general network setting at present. However, we note that in many previous works on physics-informed modeling such as [24 10 41), a co-diagonalization condition, which states that the kernel function K and the reverse of elliptic operator share the same eigenspaces, was assumed to simplify the computation. Typical examples of kernel functions satisfying the co-diagonalization assumption are the inner-product kernels on two-point homogeneous spaces [3] including spheres, real and complex projective spaces, etc. These previous works inspire us to construct the following example.
Consider the Poisson equation problem:
= f on M, u(p) Up
(25)
where the manifold M is set to be St gd++1, the d-dimensional sphere, M = = St S1 C. 2d the d-dimensional torus; is a fixed point on M; up E RR is fixed value; f is an L
Beltrami operator on M, and it is a classical function on M such that J f A is the Laplace-Beltrami
result that the problem (55) has unique solution u² E (M) (see 2 for example).
In this case, the PINN loss function of 25 and the corresponding empirical loss are in the form
(u) = =lu(p))²+ + / |Au - f13, LM((u) = lu(p)
Au(Y,) - - =, (26)
where Y, are i..dd samples on M. Define a metric on (MMM by setting
pinn (ui u2)² (p) - 22))2 + |Au] Auzl2
(27)
then L(u)- dAf (u, u²) We still denote the RKHS of NTK function K as H. and the closure of H in (M) under the metric dM as H. Since Sd a bounded smooth subspace of Rd+1 and T is a bounded smooth subspace of R22d all of the results in Section 3 still hold true for this specific problem (5). Specifically, H is contained in (M(M). hence H. the closure of H in H²2 H²MM), is well-defined. Moreover, we can show that for any bounded measurable f. the solution u² is in H:
Theorem 5.9. Suppose that Assumption 3.3 Assumption 5.2 and Assumption 3.5 are all satisfied. Then u E HNTK, hence PINN is always consistent to the problem (25 In other words, for = ((Ng), (0,4), }), we have
lim lim L (ū(())))) = 0, Ny- m0000
lim lim || - u" (1. =
(28)
Ny m0000
in probability, m. NN, T

---

Proof. See Appendix H An experimental illustration of this theorem can be found in Figure 5
Remark 5.10. The strength of this theorem lies in its ability to prove consistency even under very weak assumptions. Specifically, we only require be bounded and measurable and do not assume the typical capacity-source conditions which are adopted in many important prior works such as 46
35 37). Furthermore, the boundedness assumption of f can be relaxed in some special cases, as will be shown in an example in H.2 It is worth noting that in this example, the kernel gradient flow method is consistent while the convergence speed of PINN loss can be arbitrarily low.
45 15
Remark 5.11. Although the results in Section requires Nu + co, taking only one boundary sample p does make any trouble here, because taking N,, 1, Xi p is equivalent with taking Nu = N and X1 = =.... XN, p. both can reach the same form of the empirical loss (2))
5.2 A Negative Example
Consider the following Poisson equation problem the 1-dimensional ring:
Au= on S {p); u(p) O,
(29)
where p = == (-1.0) a fixed point on St_ This problem has unique solution u² (e
=
2122 1). (-1,1). Note that u" in H (s²) but not in (s* because its first derivative has a discontinuous point at p. In this case, we can show that the solution u' is not in H:
Proposition 5.12. The solution u" to (29 has positive distance away from the RKHS H of NTK function K, hence does not lie in H.
Moreover, the infimum of the PINN loss is infaeu L(u) L(0) = >>>> 0, and the kernel gradient flow of the PINN loss (21 is solved as vx(x) = for all x E S and t [0, oo).
The proof can be found in Appendix H.I An experimental illustration of this result is provided in Figure 6 in the appendix. Thus, by Theorem 3.7, the PINN method is inconsistent the problem (29). To be more explicit, as an immediate corollary, for any fixed time t < oc, the PINN estimator as m, N + oc, hence stays away from the PDE solution u².
Ug(() converges
Remark 5.13. The existence of the negative example (29) highlights the risks of directly applying PINNs specific problems without theoretical assessment of reliability. We notice that [29] also provided a negative example of the inconsistency of PINNs. Their example involves an elliptic equation with discontinuous coefficients: in contrast, the equation our example has continuous coefficients, while it is solved on an unusual space (S' with point removed).
6 Discussions and Conclusion
In this paper, we establish the sufficient and necessary conditions for the consistency of physics- informed kernel gradient flow and sufficiently wide PINN in general cases. The NTK framework serves as a crucial bridge between kernel methods and wide neural networks, while the character- ization of the eigenspaces of NTK remains open problem. A deeper understanding of kernel functions and RKHS will facilitate the extension of our approach to other significant problems in more general settings. On the other hand, the optimization problem of narrow networks is also difficult but important problem which this paper does not involve. We leave these valuable problems future research.
Acknowledgements
Zhuo Chen is supported in part by National Natural Science Foundation of China (Grant 12071241). Qian Lin is supported in part by National Natural Science Foundation of China (Grant 92370122, Grant 11911257) and the Beijing Natural Science Foundation (Grant Z190001).
10

---

References
[1] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
[2] Thierry Aubin. Some Nonlinear Problems in Riemannian Geometry. Springer, 1991.
[3] Douglas Azevedo and Victor S. Barbosa. Covering numbers of isotropic reproducing kernels on compact two-point homogeneous spaces. Mathemattsshhe Nachrichten, 290( (90(14)) 2458, 2017.
[4] Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco. On regularization algorithms in learning theory. Journal of complexity, 23-1): 2007.
[5] Andrea Bonfanti, Giuseppe Bruno, and Cristina Cipriani. The challenges the nonlinear regime for physics-informed neural networks. Advances in Neural Information Processing Systems, 37:41852 111188822-4881,, 2024.
[6] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge university press, 2004.
[7] L. Carvalho, J. L. Costa, J. Mourao. and G. Oliveira. The positivity of the neural tangent kernel. arXiv preprint arXiv:2404.1299 2024.
[8] Guhan Chen, Yicheng Li, and Qian Lin. On the impacts of the random initialization in the neural tangent kernel theory. Advances in Neurul Information Processing Systems, 37:35909-35944, 2024.
[9] Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical sociery, 39(1):1 1000--99,,-.11 2002.
[10] Masoumeh Dashti and Andrew M. Stuart. The bayesian approach to inverse problems. arXiv preprint arXiv:1302.6989, 2013.
[11] Nathan Doummche, Gérard Biau, and Claire Boye. Convergence and error analysis of pinns. arXiv preprint arXiv:2004.01880 2023.
[12] Rick Durrett. Proobability: Theory and Examples. Cambridge University Press, 2019.
[13] David Eric Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators, volume 120 of Cambridge Tracts in Mathematics. Cambridge University Press, 1996.
[14] Lawrence Evans, Partial Differential Equations, volume 19 of Graduate studies in mathe- matics. American Mathematical Society, 1997.
[15] Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research, 21 21(25):1-338 2020.
[16] Jean Gallier. Notes on spherical harmonics and linear representations lie groups. preprint, 2009.
[17] Weiye Gan, Yicheng Li, Qian Lin, and Zuoqiang Shi. Neural tangent kernel of neural networks with loss informed by differential operators. arXiv preprint arXiv:2503.1 10099, 2025.
[18] Yihang Gao, Yiqi Gu, and Michael Ng. Gradient descent finds the global optima of two-layer physics-informed neural networks. In International Conference on Machine Learning, pages 10676-10707. PMLR, 2023.
[19] David Gilbarg and Neil S. Trudinger. Elliptic Partial Differential Equations of Second Order, volume 224 of Grundlehren der mathematischen Wissenschaften. Springer, 1998.
[20] Boris Hanin. Random neural networks the infinite width limit as gaussian processes. The Annals of Applied Probability, 33(6A):4798 -819, 2023.

---

[21] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Convergence and
[22] Y. Jiao, Y. Lai, D. Li, X. Lu, F. Wang. Y. YANG Wang, and J. Z. Yang. A rate of convergence of physics informed neural networks for the linear second order elliptic pdes. arXiv preprint arXiv:2199001880 2021.
[23] Carlos E. Kenig. Harmonic Analysis Techniques for Second Order Elliptic Boundary Value Problems, volume 83, American Mathematical Society, 1994.
[24] B. T. Knapik, A. W. Van Der Vaart, and J. H. Van Zanten, Bayesian inverse problems with gaussian priors. The Annals of Statistics, 39(5):2626-266 2011.
[25] Jianfa Lai, Rui Chen, Manyun Xu, and Qian Lin. Generalization ability of wide neural networks on https://arxiv. org/abs/2302.05533..
[26] Ye Li, Song-Cau Chen, and Sheng-Jun Huang. Implicit stochastic gradient descent for train- ing physics-informed neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, No. 7, pages 8692-8700, 2023.
[27] Yichen Li, Zixiong Yu, Guhan Chen, and Qian Lin. Statistical optimality of deep wide neural networks.
0266077 2023.
[28] Junhong Lin and Volkan Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research, 21(147):1-63, 2020.
[29] Tao Luo and Qiyuan Zhou On residual minimization for pdes: Failure of pinn, modified equation, and implicit bias. arXiv preprint arXiv:2310.18201 2023.
[30] S. Mishra and R. Molinaro. Estimates the generalization error of physics informed neural networks (pinns) for approximating a class of inverse problems for pdes. IMA Journal of Numerical Analysis, 43(1):1-43, 2023.
[31] Richard Nickl, Sara Van De Geer, and Sven Wang. Convergence rates for penalized least squares estimators in pde constrained regression problems. SIAM/ASA Journal on Uncertainty Quantification, 8(1):374 413, 2020.
[32] Loucas Pillaud-Vivve Vivien and Alessandro Rudi. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31, 2018
[33] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informen neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 278:686-707, 2019.
[34] Y. Shin, J. Darbon, and G.E. Karniadakis. On the convergence of physics informed neural net- works for linear second-orde elliptic and parabolic type pdes. arXiv preprint :00::22000001000 2020.
[35] Steve Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive Approximation, 2622)
[36] Ingo Steinwart and Andreas Christmann. Support Vector Machines. Springer, 2008.
[37] Ingo Steinwart, Don Hush, and Clint Scovel. Optimal rates for regularized least squares regression. COLT, pages 79-93, 2009.
[38] Ingo Steinwart and Clint Scove. On the interaction between measures, kernels, and rkhss. Constructive Approximation, 35:363-417, 2012.
[39] Roman Vershynin. Hige--Dmensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018
12

---

[40] Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Commutational Physics. 449:110768, 2022.
[41] H. Wong, W. Wu, Liu, and Y. Lu. Benign overfitting fixed dimension via physics-informed learning with smooth inductive bias. arXiv preprint arXiv:2406.09194, 2024.
[42] Xianliang Xu, Ting Du, Wang Kong. Bin Shan, Ye Li, and Zhongyi Huang. Convergence of implicit gradient descent for training two-layer physics-informed neural networks. arXiv preprint arXiv:2407.02877 2024.
[43] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian pro- cess behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019.
[44] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315 2007.
[45] Haobo Zhang, Yichen Li, and Qian Lin. On the optimality of misspecified spectral algorithms. Journal of Machine Learning Research, 25(188):1-50, 2024.
[46] Haobo Zhang, Yichen Li, Weihao Lu, and Qian Lin. On the optimality of misspecified kernel ridge regression. In International Conference of Machine Learning, 2023.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction sections, we provide comprehensive summary of our main results, and list the abstract of our technical contributions as supplement.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Some limitations of our work are discussed in Section 5.1 and
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All the assumptions required are listed in the paper, and detailed proofs are demonstrated in the appendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of our experiments have been discussed Appendix !
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Our experiments are of small scale, and are very easy to reproduce personal computers. Besides, these experimental results are provided only for the purpose of illustrat- ing our theoretical results. Since we have provided all the details of these experiments, think unnecessary to release the codes.
Justification:
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The design of the experiments, including the structure and the initialization of the network, the generation of samples, are provided in the paper. More details of our experiments are discussed in Appendix!!
Justification:
7. Experiment statistical significance
14

---

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The only purpose of the experiments is to illustrate the theoretical results in this paper, hence think unnecessary report the statistical significance.
Justification:
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
Such information is provided in Appendix !
9. Code ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/ps//ppprr...cc//pp//////a
Answer: [Yes]
Justification: The research in this paper conforms with the NeurlPS Code Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This paper does not use existing assets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and research with human subjects
15

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: In the writing of this paper, the LLMs are used only for vocabulary and grammar checking.
16

---

A Smoothness of RKHS
In this section, we prove an important theorem, which verifies the well-definedness of the physics- informed kernel gradient flow estimator (13). the population gradient flow (21 and the space H defined in Definition 3.6
Theorem A.1. (Smoothness of RKHS) Assume that Assumption 3.3 and Assumption 5.2 are satisfied. Then any u H. also in (?));
Moreover, if the RKHS expansion of u H. is given by u(x) Ei= us (x).(x)), then can be twice differentiated tern by term; to be more specific,
V'u(a) E (V 1= 1,2.
(30)
The proof of this theorem is divided into the following series of lemmata: Lemma A.2. The eigenfunctions o; of K are (S2)-
Proof. It is direct consequence of the equality
Φc(x) Xi L. K(x, ((x,)(()) ())))))
1 1
(31)
Lemma A.3. oy (.,,) and 02 uius
K(, y) are in H for any fixed E and i = I....,d d.
Proof. Let 22 ....... 0, 1, 0, . 0) Rd with the i-th component be 1 and the rest to be 0. We
first show that the difference quotient
(h) K(,9 + hews) - K(., w)
(32)
h
a Cauchy sequence in 7 as h Y 0.
Recall that by the definition of RKHS, for any V1. E S2
(K(,))), K(,,22) = K K(y1,32),
(33)
hence for any h1, h2 > O, we have
(Ai(hi), (h2))
h1 (K + 11 + 1222)) K(nn + (2),,12) - K(21, 82 + 1222)) + K K(y1, 12)) (34)
+0,, Oy, K K 322
h1 h2 > 0. Thus,
[11(h1)) - (,
(,)),, - 2(1,(h)) (x(h2) + (((2)) + 0 (35)
as h1, h2 > O. This implies that (h) a Cauchy sequence in H. hence has a limit A, E H. A, is also the 2 limit of (h) since
(h) - A1lla s (1,(h) - -illl-
(36)
However, (h) converges to 0g K(:y) pointwisely. Thus, o, K(. v) = A, E H.
The proof for v.v, K(, ((.y) similar.
Lemma A.4. For any E Ω the sequences
(,,0y and VT (5(
(37)
are in 12
17

---

Moreover. let = 0,00 E Rd with the i-th component to be and the rest to be 0.
Then
Vicdely + -1y) (Tossl)) v V.day,bily)
(38)
h
and
Vi,og, + ht) vareg, v (, in in
(39)
h
Proof. Note that
(y(h) K(x,y + huz) - K(x,)) v)
Vy +b)) - (Tsi(()).
(40)
h
By Lemma A.3 A (h) converges in H. In other words, the sequence
Voolyy + hy) (voo(v)
(41)
converges in (2, Since Vrrd,, the termwise limit of the above sequence, then
Viroi + h2) vTrdi()) (rr0g, in ?.
(42)
h
Likewise we have
V, + h4) (Ar0p,0r(1)
(x(v) Mswe ( in in
(43)
h
As direct consequences of the above lemma, we have the following differentiation properties of kernel function and functions in the RKHS of K:
Corollary A.5. For any multi-index 8 with 1
05 K2 10
(44)
converges absolutely and uniformly.
Corollary A.6. For any u twice continuously differentiable. Moreover if the RKHS expansion of is
u(x) Σ (oTTeee(a), {mi) EP,
(45)
1=0
then for any multi-index with |a| 2.
2 u (
0C((z)= = Σ
(46)
1=0
converges absolutely and uniformly. Corollary A.7. For any E H,
| S Clluu|
(47)
Proof. Suppose that
u(x)) Σ Vedse()), {uit EP.
(48)
i=0
By direct computations, for any x E Ω2 have
2c
[u(x)] < u. (m s 11lli (ooo()) = ||l| v((x,)),
(49)
18

---

Or, (x)] s Σ
V Nr (s(()) - |lllle 1I
= 11ull Ooy 0e, ((x, (50)
1=0
and
8 myze u(x)] -
Σ
00 01(x))
1=0
(51)
Sluell1
(x))ll
||ul|ze O Øezza V 2(x,
The above results complete the proof of Theorem A.I
B Proof of Lemma
In this section, we analyze the difference between the empirical gradient flow (3
No
Nj
dt 0z(x)
Σ K(x,
-.))
Σ C,K(,,))) (Cū(())) (C
5,) (52)
No
N₂
and the population gradient flow
21
Ji [v)(w)= 2 / on
d
K (xx,E) ve(E)de -2 / LyK(,,v) (Cor(v) f1))ddy.
(53)
Lemma B.1. As N2 oo, we have
N2
2
C,K(,,)))) (CO,(Y) f(x)) E2) -2 2 (gK(,,)) (Ca,(o) f()))dy
Σ
sup =EN,,E(O,TT
N
==1
-o -0 (
1 log N N
(54)
with high probability O(N7'). where the conventions O hide the terms involving Q, K, and T.
Proof. Define
U:(1) / (x:)((ss) F5(1) /'
((3)
(55)
then
Lū,(x) = (
d ca (x)ds ds
Σ
No
N₁ Σ
N (c, (56)
N.
We make the following decomposition:
N;
2
C (x, Y3) (Li+(Y)) -() = +T2 T3 T4 T3.
(57)
N
where
2 N₁ C,K,,,,)),,,,(
Ti
(58)
N₂
2 Σ JcKKEeYY),CC,,,,,,,Y. Y.)
T2 N3 j#k
(59)
19

---

2 N₁
T3
(60)
N7 j=1
N₂
2
T4
Σ
(x, (3)(()))))))))
(61)
j=1
2 N
T. = Nj Σ (2,K(,
(62)
For brevity, we only prove the estimation for T4- The proofs for T1, T2. T3 and T3 are similar. is easy that by strong law large number, for any fixed x E 9
T4 - -2 2/ (x, (y)dy
(63)
almost surely 2, goes infinity. Next, show that the convergence is actually uniform with respect to S.
By assumption, Ω Ω Since 52 is bounded region, the random variables Ly K ((x, (Y)) sub-exponential obviously. Then, by Bernstein's 's incquality (see Section 2.8 of (9)), we have the following estimation:
(S2 is bounded. And by Theorem 5.4 L y) Lipschitz for (z, y)
N Σ (T,K(,Yy))(0, + / (,, > n )n) N
S 2e -e--mma(NNn" (64)
P
where the constant e >> 0 depends only on Ω, k and
Next, we apply the g-net argument to æ E Ω. Without loss generality, assume that Ω is contained in the cube [0, Djd For any n E N, we define
N₂
2, = -....,,++l.. i I...., d} C [0, DI
(65)
and then the canonical number of Ng is (n + 1)², and
2
La ())(x) +2 (((lai In
max PEN,
N
(66)
> - 2(n + 1)ae eemm( Myn)
Moreover, for any Ωt there exists x' Ng such that |x- x'1 ed
For any x, x' E Ω2 such that Jx 2'1 < ed, by applying Bernstein's 's inequality again, we obtain that
2 N
2 N N
P
sup
K,K
Σ 1=1
K,, (KK(a()))f(Y)))
sn
N 1=1
1x-z' Sed
2d' d N₂
>P
Σ f(Y;) 51
N Ø=1
N
(67)
J(Y,) -2 2
< 7
S 11
2P
N 1=1
2c'
ed
N S(Y)) - f
n
=P
N
Ed!
21 - 2e c" min (onn((Ny N 2d Nyn//=1 /
if we select 77, such that f2 fl Zergu and the constant d' > 0 depends only Ω, K and f.
20

---

We also have
2 S/8 Ly (x, (z;
K ())))d S c²
(68)
where e" > depends only on Ω K and f.
Combining the estimations above together yields
2
K XX,((() +2 / (,((1))) s (2 + d")n)
I
sup xel2
2 N
K, +2 / (,K( 2111 2m))
21 PP
DEA,
2
(69)
N
N N₁
-P
sup |2-2'| Kedh
Σ ((((),))))()) >1
N
- 2(D/s 1)6e ceme-mmmnnppp (Ny0 ,y)) 2c-d" (in 1e" Nynte").
Finally, we select 2dd x Ny and n²² x log Nv, then with high probability 1 1 ((N ), we have Nr
2
N Σ
(,((,, +2 / L,K(s, (0 log N₁ N₂
=o(v
sup sEN
La,
(70)
j=1
Likewise, we also have the following estimation for Ti, T2, T3 and Ty:
2 N N
sup zEN
Σ
X,
+2 LrKly, X.),,
==1
(71)
log No N₃
Σ C,C,,)))
(z.( (x)
sup zel
N1 ISiEN,,jAk
(72)
(v
log 1 N₂
0
k 1,2,..,,
2 N
log N₂
sup zEi
L K(, (,,,,,)),,,,KKK
=0
(73)
N2
N
N
log Ny N₂
sup =EN
C,K(,, =0
(74)
N
with probability 1 1- O ) We also note that
[ C. K,((1,)) -CLū - fyy)dd
No
N₁ Σ
(75)
= / (,(()))))
Σ
C.C.KK(,,Y),, f(8) -f())
dy.
N₂
N₂
j=1
21

---

hence
N₁ b (xr (Li((V)) - -e,) +2 / (x v) (Cirly) - f(u))dy
2
==1
E U.S U.
(V logNs N,
1 N
Σ Fj o ((
N1
log,N,
(v
log log.Ns
+
dy.
Nj
Ny
(76)
By Cauchy's inequality, and by the definition of U, and Fj. have
N
(CG,(Y)) - f(X)) -x) +2 / (oK(z.V) (Cūr(v) - f(y))dy
(x)(x.
N
f o
(v
N1
1
N.
N
(X,)) +
Σ - - =2,}² N₂
dsdy
0
N₂
N ==1
(1
2
+0
log N₁
Ns
=0
log N LoggNN N₂
f ((, +
(Vo N N
2
(77)
Since [(u) decreases along the gradient flow, then [(u(t)) S [(u(O)) for any t E j0, T). hence
J. L(d))ds s Ti(O)).
(78)
In conclusion, with probability -(( have
2 N
C,,(,, Y2) (CC,,(CC)))))))) (()))))))(()))) -2,) +2 1 CyK(z, -+)) (Cer(v) (w))ddy
sup *EN,CE(O,T
N
= =0 (
log N₂ N₂
(79)
Likewise, we also have
Lemma B.2. As No. oo, we have
2
Σ (x, (x(X,) +2 / K(x,g) ac(E)de =0
log Nu. N₂
sup (0.7)
(80)
N
on
with high probability 1 O(N
Proof. The proof is similar with that of Lemma B.1
Lemma B.3. With high probability 1 O(N N '), we have
T log N2 N₂
log N
dpinn (u, vv) 0
+
(81)
No.
for any t e [].
22

---

Proof. Take ût 22. Then by Lemma B.1 and Lemma B.2 we have
(,) - / K(x,E) (e( 2 / ((, (ue(r)dn + R.
(82)
where
R= 200 (Vr log Nu
+0 (1T
log N₂ N₂
(83)
Na
with probability 1 ) O(N
Define ((t) = dpinn (ūt, Re), then
W(1)² ( 12)2 ./ lur (x)(x)) I |Cur (uvi(w)) (v))2
(84)
and
T ( -22 R (1 wi(x))d / (wels)dd))
(85)
where
2 Jn Jon K(r, (tx) + / on
(w) dxdd
(86)
J. I 1)
(w) (yC
Assume that the RKHS expansion of wi is
(x(x)) Σ 1 (r),(t),
(87)
1=0
and recall that
K(I,0) = (z)
(88)
then by computation, we obtain that
Q= Σ (1
wtdroilo)de I wt (f>: )² 0.
(89)
5 on
Thus, by Cauchy inequality,
d dt
(W(t)) s R (1 we(x)dd + ( Lurleldy) $ 2R)()),
(90)
then
d ((t) SR
(91)
di
Therefore, for any E [0,T], TJ. we have
W(t) = ((t) ((0)= = ( ds
(()dss So T
log N, N₂
Tog N₂ N.
+ V
(92)
Proof. (of Lemma
4.1
Note that
IL(,,) L(vz) = Idpinn (ūe, 21²²2 2 dpinn( u ?
(ddpinn (ih,, u²) dpinn (Vr, v² ))²
(93)
2dpinn (Ve, u² (dpinn (We, 11 Apina (Ve, u²)
< dyiin (De, ) + 2dpin (Ur, u dpinn (Se, Ve).
23

---

Since L(v)) = dpinn (U,, u²) is decreasing, then
dpinn (vr, u²) I dpinn (VD, u²) = dpinn (0,u²) v |sp.
(94)
Thus, by Lemma B.3
with great probability
+ +Nx O
we have
[L(as) - L(ve)] = ( T
log N
+ v
log Noy No
(95)
N
for all E [0,
C Proof of Lemma 4.2
In this section, we consider the population gradient flow 21:
dt [x) -2 f (xx,E) vx(E)d - 2 S (ar,), (Lvx(v) (1)))
d
(96)
C.1 Diagonalized representation of population gradient flow Assume that the expansion of v in RKHS
(97)
then along the gradient flow (1). the evolution ODE of the coefficients i; vz(t) is
dt v1(() -2 / Vidae (((t)//00)) 2 2/6 VirCo Σ o,(()) Ty -r)
d
j=0
(98)
= - (Sig + (ty)uu(t) + Ris
j=0
where
Tyy= Sij - / VAAooio Tij - / VAX,LOLLOS, R R- VA.Co.f
(99)
The infinite infinite-dimensiee dimensional symmetric matrices Sig and Túş are the Gram matrices the following self-adjoint operators on H:
S: H +
Se Σ
StzUj
V Viie.
(100)
i=0
/=0
T: H +
To
Σ
Σ
Tijuz
Virai.
(101)
1=0
The population gradient flow
21 can be rewritten
d
di 2t = 2(S +TS+))) +
(102)
where
20
R Σ R
(103)
1=0
Lemma C.I. S and T are compact.
24

---

Proof. For any E H, let w = Sv, then
2
00
8
(ls Vaidselos)
2
1w: 2
Σ
1ig0y
Σ
Sij
1101 Σ
==0
1=0
j=0
1=0
$
||0
Rt.
/
2503
11e11/4
Jon
/
A1o
Σ
250
(104)
00
05 j=0
/ K(x, x))x.
||ا
X,O
Jen
Thus, maps the infiiite-dimensionall ball {ur E H: ||l1 S B} into the anfinit-ddimensinn cube
QB {w H w, Ba,},
(105)
where
1 K(x, x)dz / Aoo,
a7
(106)
Note that
Σ a (1o ((x,a)dx)) < 00,
0
(107)
i=0
hence {a:} E 1?.
suffices to show that any sequence {w* J C Qn has a convergent sequence in H.
Since {wo) } is contained in [-Ban, Bao], it has subsequence {w) (06)) such that wo 19
Wi as
j -- co. Then we select subsequence of [w] denoted w j - oc. Iteratively, we select subsequence k²) from kf) such that w/ (u(()) + wy as j - 00.
k () such that wi k² (t)
as
Note that |wi| S Bug, hence {wi} E 12, Define
Σ wi v Tich H.
(108)
Then we claim that
W k. (1)
w 1 - 0 as l + co.
(109)
In fact, for any & > 0, we can decompose
|/ - w H into two parts:
N
||l (1)
wl
11(0 [
w:)2 2 +
(w
(1) w:
(110)
1=0
i=++1
where N E N is selected such that
Σ of KE
(111)
1=N+1
Then we obtaín that
2 w:)) 2
Σ
(w, 1.
Σ
(ww, #2)02 + wτ < 4B2
Σ
a <B
(112)
i=++1
N=N+1
i= N1
Thus,
lim Iw k (1) w wl11 < 4B2g. 1+000
(113)
Since e is arbitrary, we conclude that
lim 1| (()) - -wll 0. 100
(114)
25

---

Thus, S+T is self-adjoint compact operator on H. By Riezs-Schauder theory, S+ has eigenvalues No 21 2 ... 2 0. μ4; 0 as + oo, and { the corresponding eigenbasis.
Under the new orthonormal basis {v/)}. let üg be the coefficients of expansion of et:
vx(x) = Σ (t))))))).
(115)
Then under the basis {v:). the gradient flow 102 can be rewritten as
d (() ())x0((() + R.,
(116)
dt
where {Ra} E 1e is determined by
R = Σ (1),())
Σ (x).
(117)
Thus, for each of is increasing along the gradient flow
(1
C.2 Convex optimization
Lemma C.2. [[V[(v)]]x - as - oo.
Proof. It suffices to show that L(ve) is a convex function with respect to t E [0, oo). The convexity directly checked by
d d12
L(ve)
d
Σ
ÖL
(vz)
2
--2
OL
(vr)
0LL
(v))
dl
aldv;
1=0
==
ØL
0L
-
(v:)
(wx)
Ov
Øt
1=0
OL (v:)
0
2)
(118)
Σ
(vx)
0%
Oz,
1=0
1=0
80
8
OL
LLL
0L
4
(v:)
(vt)
Oug (vz)
0o,00;
1=0
2
Along the gradient flow, L is non non-increasing, hence L converges to some constant A O. It sufficess prove that A inf L(v).
Proof. (of Lemma 4.2)
Assume otherwise. Then by the convexity of L, for any > O, the set {u E H: L(v) < A} must be contained in one side of the tangent hyperplane u(t):
(w, ((v))) < (v.. vL(v)) Ve E {w E H. L(w) < A}.
(119)
t. Then we can fix one element w E (w E H inf L(v) < L(w) < A}. and denote its coefficient series with respect to the basis {v)) as {w; Hence
By assumption, I(w)) 2 A > inf.em Luv) for any
OL (wt) 00;
aL 06:
0z
(v.)
(120)
which implies that
8
dū,
20
OL
di U1
w 12 = 2
Σ
(1)(1)=
wi)
2
(o:(1)
w.))
(vz) S 0.
(121)
dl
1=0
26

---

Thus, along the gradient flow, the RKHS norm of u(t) is bounded by
1Ivel S 1/0 lie.
(122)
The equation (16) implies that (t) is monotonic. Thus, l increasingly converges to some C 2 0. In other words,
lim
6111==
(123)
Let V, = lim ü.(t). By monotone convergence theorem, we have
20
V
Σ Viv E H.
|V = Σ
V.2
lim 100
Σ i,(t)²
C <<<0 00;
(124)
hence
0 Vl + O.
(125)
By Lemma
C.2 and the continuity of VL, we obtain that
|V = lim [[ = 1-00
(126)
Then V is in fact a minimizer of L by convexity, hence L(e,) L(V) = 0. This contradicts with the assumption that L(u)) 2 inf L(u) 2 This completes our proof.
D Norm control
Proof. (of Lemma
3.9
We decompose u u² into + w. where
Lv = Lu
in Ωi
Lw =0 0
in S2;
(127)
==
on an
u
on OΩ.
By the standard elliptic PDE theory, the operator C-1; L2(( + H1 is bounded, hence
|ellia I Cl|Cul12: = C/ f
(128)
where the constant > 0 depends only on and
For the estimation of w. we first apply Green's 's representation formula to w and obtain
w(y) = f K(x, (((x)))(z)dd K(x,8) (AV
(129)
where A (asz) is the coefficient matrix T,, G the Green's function of on Ω2 and v(x) the outer unit normal vector of 02 at is well-known that (see Theorem 1.2.8 of [3] and its remarks for example)
K(x, v) (AV, v(x) 20, Ve E 02, y Ω
(130)
and
|,,((,,)) v) S Clr- 21²- for some C > depending only on Z and L. Therefore.
(131)
K((,,)) g)1 < JAV G( x) < ||||| VV v) & Clx y1-
(132)
and
[ 2 = I ( VK(x,u) / /K((,, (((r,))((rdde dy
(133)
< / ( K(I,,
((x,))dx v)da I Kla, K(a,x)a(x)) dy.
Note that the function
n(v) = 80
y)
(134)
27

---

is the solution to the boundary value problem
Ln =
in 92;
(135)
17 =
an,
then by the maximum principle, |n/ I 1. Thus,
/ u/ w(yy)2y v [. f K(
- / Jmn
(Jo Clz v/ "1" u(z)
(136)
$ C
dx,
Jan
where C" > depends only on 92 and L. In conclusion, have
I|u u (())) & ||ll + [| S (IILu (illau- + -IllB( = L(u( (137)
for some C" > 0 depending only on f and L.
E Proof of Theorem 3.12
Lemma E.1. uf E then
IIu- 1u-u411 u=11 |VL(u)l> L(u).
(138)
Proof. When u u", then V )1 L(u) =
When u * u², define R = ||u - u" u²| and w = (u - u RR. Let v(r) = L(u" + rw), then
(0) I(u") vb Rt) [(u), and
(r) V L(u" + rw) (VL(u* + rw), w)n-
(139)
Since L is convex, then also convex on [0, Rj. Moreover, since vi(r reaches its minimum at r = 0, then (((rr is increasing. Thus,
[(u) (RR) (00)= {aa (((r)dr s (R) (R) R(VT(u), S R[VL(u))
(140)
Lemma F.2. Along the gradient flow va defined by (1))
we have
C
L(ve) S
(141)
t
for some constant C >> 0 depending only on Ω L. f and K.
Proof. Without loss of generality, assume that n² * 0. Let T((o) T((0) M, then M depends on S2, L and f. Along the gradient flow 21 the loss T(1))) is decreasing. By Theorem A.1 u" ( Then by Lemma A.7, we obtain that
||ve u" 1|
I I|ue- u" lccc & &'T(M))
< ('
C'M
(142)
for some C' > 0 depending only on and K.
Finally, by Lemma E.I we have
d L(vi)
20
Σ
(vx)
di
Oo,
(143)
2 (C² I(u)2.
hence
L(vo) L(vi) f L(ve)2 dt L(viddt (2'
(144)
28

---

Proof. (of Theorem 3.12 Lemma
The conclusion is immediately obtained by combining Lemma and
Proof of Theorem 5.4
In the rest of this paper, we will discuss the neural tangent kernel KNT, its RKHS HNTK and the network (g(t)- For brevity and with no ambiguity, the following context, we denote the neural
tangent kernel as K instead of K KNT, and denote the RKHS of NTK as H instead of HNTKK
We will first prove Theorem 5.4 for shallow networks with only one hidden layer and and then provide proofs for deep neural networks in Section F3
1) Section
When = 1, the network (5) has the following form:
1
(x)
Aga(WN x + BB) + D,
(145)
2m
where the parameters are initialized in the following way: Aktm = - As. W WR. Ba+m Bg, = 0, and Ag. WK,,, , Bg N(0, 1), .,..., m, l 1...., dra initialized as i.d. standard Gaussian random variables.
F1
Convergence at initial time
F.1.1
Pointwise convergence
Since bounded domain, we assume that 5 is contained in the cube Q(R) = [-R,, Rja.
Lemma F.1. For each z,y Q(R) = [- R) Koçoy (z, y) converges K(I, y) almost surely.
Proof. By direct computation, we have
Ke(oy (2, v) =1 o oW . x B&) oWW..1 B&)
(146)
AL.(1 + (2, . x Ba) o' + B&)
with the parameters initialized as i.i.d. standard Gaussian random variables. Thus, by strong law of large number (see Section 2.4 of 12 for example), it converges almost surely to
K(x, v) =1+ E (W1 x B1) (W1 y BB)] + (1 + 1 Afo' (W1 x + Bi) of (W1 y + Bi)
(147)
Lemma F.2. The limit kernel K(r,, y) is C2 with respect to Q(R). Thus, L,K(r, exists and continuous for (z, y) Q(R) x Q()..
Proof. For brevity, we only prove that V, (r, y) exists and is continuous with respect to v. y. The proofs for the second derivatives V K (ar, y) are similar.
Note that K K(x v) = 1 + G(æ, 3) + (x, y). where
G(x, v) E [ I + B1) oWW y Bi)],
(148)
(x, = (1 (1 (x. F A3 + Bi) o (W, y B1)]
(149)
first prove that the function G( 3) is C with respect to z. Denote v = OL..,.... 0) E Rd By mean value theorem,
( (x tv) Bi) (W1 Bi)
Win [W (x + sv) + B1)
(150)
29

---

for some E [0, Since o' is polynomially bounded, then for fixed the function
o ( (x + te) + B1) oW + B1) a (Wi y Bi) S (WW,, B1)
(151)
for some polynomial g, which is integrable. By dominant convergence theorem, we have
G(æ + tu, v) G(x,y)
lim
o( (x + tv) + B1) - o(W1 . I x+ Bi) B1) oW w.y.yy + Bi)
-E
lim t-0
(152)
L
0 a (W1 x+ + Bi) (W, y + Bi)),
-E( dm
which is continuous with respect to x, y.. other words, G y) has a continuous partial derivative in the direction Zi. Likewise, continuous partial derivatives in other directions also exist. Thus, G(v,, v) differential with respect to x, and
VzG(z, y) = E(V (W - I + B1) oWW1- y B1))
(153)
continuous with respect to x;
Likewise, (x, y) is continuously differentiable for x as well. Thus, K(x,v) is continuously differentiable for æ.
Lemma F3. For each v. E Ω V (aroo y). Keto (sr, y) converge in probability V,k(w,)) and V K v), respectively. Thus, L, Kot (x, y) converges almost surely C (x, 9).
Proof. By strong law of large number,
V (o y) =1
V. (W x B&) oWW y B&)
m
m
1 + (z, (z (We x Bx)= o' 2WW . y + Bk)
(154)
m
k=1
Aly a d'WW x BB) o' d'(Wn: y Ba)
m
converges almost surely
V VzK(H,y)
=1 + E (W1 + BI) o(W y B1)]
+ (1+ (z, y) E A o (W1 + +B) o' (W1 y + B1)]
22
(155)
+ yE [Afo'(M x + B1) dW y -yBB).
Proofs for the second derivatives are similar.
So far, we have finished the prove for the existence of K K(x, y), C. Ka,, y) and the convergence of Kero) (x, y), C. Ke(oy( (I. y) to K(x, y). L. (x, y), respectively. For Ly ((x, v) and C. v). by following similar discussion, we can also obtain
Lemma F4. (x y) and Ly y) both exist and are continuous with respect to (x, y) E Q(R) x Q(R). And for fixed x,y, Cy K (x, y) and Ly Kocoy (x, y) converge almost surely to C (z. y) and CyCyK(e, y), respectively.
F.1.2 Uniform convergence at initial time
Recall that we assume Ω c Q(R) = [- Rl.d. We will show that the convergence of Keçoy (x. y) and its derivatives fact uniform for x. y E Q(R). The proof is based on 5 s-net argument.
For N N. set 2R/N and place dd points in the cube Q [-R,
No (=
2ig R (-1+
2id N
: is 1, ,....,y N, k = 1, I....,d d) (156)
a)
N
30

---

For any 2 E Q(R), there exists some No such that x 2/ S <vv..
Again, denote Ko(o) (I, (m v) (x, and K(x; y) = 1 G(a(() (o. where
1
(m, =
oWW x Bx) o( y Bx),
(157)
kml
G( v) E lo (W1 I + B1) oWW B1)):
(158)
Hm(x, = A (1 + (z, v) (W x BB) o ( y + Ba),
(159)
H(I,y) = (1 (z, v) E A (W1 x + B1) of (WW y + B1)]
(160)
For any y E QR),, we select 2, E No such that |x = s sevd, |y w| S sevd. Using triangular inequality, we obtain that
[G, y) - Gzz, v)1 & (G, y) Gm w)ll + [G(,, w) G(x, 8) + Gm (2, w) - G(z, w)
(161)
[H, y) H (x, v)1 & Hmm (2, v) (2 w)) - [2, w) H (z, v) H (z, w) (2, w)).
(162)
The following lemmata provide the estimations for the terms on the right hand side.
Lemma F.5. Define the event
(|A+). W a. [&| S M, I....., m, l 1,..., d].
(163)
where M /3 log m. Then P(B) 2 1 (d 2) 1/2,
Proof. If Z (0, 1), then classical Gaussian tail bound gives
2e M2 /2
P( 2 M) S
S m 332
(164)
M
hence
P(B) 2 1- mm(d+2)P((1) N M) 2 1- 1-d1+++ 22m 1/2
(165)
Lemma F... Conditioning on the event B, we have
G y) Gm, w)l C₁ (log p+ b+
(166)
Hm (x. y) (m w)ll < Ci (logmm)tt where the constant C₁ > 0 depends only on d, R and a.
(167)
Proof. Recall that we assume Ô c ((R)) = | R, Rj.. Conditioning on B, for any x E Q(R), we have
d
We x Bxl <
Σ
Wa,1 |xl + |Bal < dRM + M.
(168)
By Assumption 5.2 (a(r) S C² + Ir()), (r)) (11 Ir|) for some C' 0. Then
lo(W x + B&)l S C'((1 1)M))).
(169)
hence
WW + Bx]l [Ws o d'(Wa + Bb)ll S || lo x Bx)l
(170)
<VMM C² ((dR + 1)M) < M++
31

---

where C" depends only R and d. Thus, the Lipschitz constant of Gm is bounded by
sup k.)
[V = sup
(W x BR)o + B&)l
3.
m
S sup
[V x B.)o(W + +B))1l
z.y.k
22 sup
| (oWw: x x+ Ba y B&))ll
(171)
a.y.k
22sup IV o x B&)ll sup
( y B&)l
y.k
MPt1 C1 + (ddR + 1)M)P)
C" MP++
for some C depending only on d, R and a. This yields the conclusion. The proof for Hm similar.
Lemma F.7. We have
GG w) G(z, g)l C2s, [ w) (z, v) S C2e,
(172)
(173)
where C2 > O depends only on d and a.
Proof. We have proved in Lemma F3 that V,G(z, y) and Gz y) is continuous, hence G(x, y) Lipschitz on Q(R). and by the explicit expression of VG, the Lipschitz constant Lg depends only on a. Since | - =| S vds, ly w| < dds, we have
[G(z, w) Grr, v) S (G(,,w) G(z, v\1 + (G(,, v) G(r,,)) S 2hovde..
(174)
The proof for H (2, y) is similar.
Lemma F.8. We have
sup s,WEN,
(o. s Csp
log m m
2 2
2R
2d
m
1-2
(175)
8
log
(2
2d
P
sup i,WEN,
(Hm(z, ()) H(2, w)l S Cay
2
m
(176)
m
where C3 > O depends only on d, R and a.
Proof.
For x,y E Q(R), the random variable oW1 x + B1) o(W1 B1) is polynomially bounded:
1o(W + Bi) oWW, y B,)l s (1 (R)
Wi,1 + B1)
(177)
hence is clearly sub-exponential, and its sub-exponential norm has an upper bound > 0 depending only and d. Thus, by Bernstein's inequality (Section 2.8 of (39], ) have
f 52 = =)))
IP (| (2, - G(x,, x/ 2 r) < 2exp (-d"m m' min
(178)
where c" > 0 is universal constant.
Note that |N21 (2R/e)*, hence
IP sup (m,(2, w) G(z,w()) <r)) 1 (2 L.WEN,
22 ())) (179)
72
exp -c"m
min
32

---

V (1+4d)o log m we obtain m
By taking
log m
2R
2d
sup
[G,(2, w) G(z, w)l < Cay
1 2
m
22
(180)
2.WEN,
m
8
The proof for Hm and H is similar, and we also have
log m
(2R
2d
sup s.WEN.
[mm(,, - H(, w)l < Cap
2 1 -
m71
--20
(181)
m
Combining the above lemmata together, we obtain
log
sup ())
(m, G(r, v)1 S (t C2e + Csp
(182)
m
log
sup (R)
[mm v) - H (z, f) - (i C2E + Cap
(183)
m
with probability 1 2 (2R) 2d (
m 1/2222 (d + 2)m - 1/2, By taking s = 1 L/m, we have
log
sup =
(Gm(z.)) G(z, 3)1 S CV
(184)
m
log m
sup x.yE((R)
Hm (2, y) H(x 8)1 s CV
(185)
with probability 1 - cm 1/2, where the constants C' and depend only on d, R and Since Ω QR)), we obtain
Lemma F.9.
log m
sup (o1o) K(x, v) & CV
(186)
z.yen
m
for some constant C >> O depending only on Q and a, with probability - O(m 1/2).
The proof for the derivatives of Ko(o) and K is similar. Following similar discussion above, it is easy to prove
Lemma F.10.
sup (or) (x, L,K(z, v) SV log m 2.yeft
(187)
log m
sup (x,) L,K(x,))) $ sCv
(188)
zyyefl
sup Ko L2L, v)1 S 3 log m
(189)
z.ven
for some constant depending only Q and with probability
1 mm
33

---

F.2 Convergence during training
Lemma F.11. Conditioning on the event defined in Definition the parameters of the network satisfies:
F.5 along the training procedure,
(log (log m
sup sup (0y(t) - 0 =E(0.7]
(190)
v m
for 0 -0 Ak. 3 or Bg, and the constant p 2 is defined Assumption 5.2
Here, the convention
O hides the constants depending on d, T, S2 o, f and C.
Proof. Recall that the network is be
1
(o())) (x)) V2
(c())o(ax(t;2))) D()),
(191)
where or (l, Wx(1) - (t).). Then
(i)()(
(an((t(()))
V22m k=1
2m
A
ai W
(192)
2m
k=1
i,5=
2m
As
by WK (ax,(()) co((acl,,2))
2m
k=1
where b, = b, + Σ
i=1 O,agg.
The gradient flow can be explicitly computed follows:
d
2
A
(ax Xx)) (aco
dt
N
2m
(193)
N₂
2
(C) -fxx))
N
2m
2 N.
d
1
Wh1
A X, (o X,)) (X,)
dt
"N,,
/2m
i=1
N₂
au WaW (((nntt, + bio (an(t.,z)) (V)) - -VV))
A
N j=1
2m
i=1
N
d
A&
Σ
Omn Wk.m Wk, Yy,o" Yy.oa" (ax(t, (Caon)) -X3))f(X)))
N
22m
J=1
m,
N
A
Σ
bm Wkmm (axo"(a((t,))))) (Ciocs) (Y₂) -())))(()))
N
22m
m=
N
2
Av (cY (ax(1, (()))) (Lūoto) (Yz) ((X))),
(CVs,,o'(ax((,,
N
22m
(194)
34

---

d
2
Na
1
Bk
(ax(tt, (Xi))
dl
Nu
22m
N
2
1
Ak
Σ
amn
Wk kmm Wh W (ax(t (Cioco) (X3) ,(Yr))
2m
(m,n=
2
1
Ak
Σ
bm
(" Yj)) + e (au(t, (x)) f(Y,)).
N j=1
2m
(195)
a E QRR) = | R, nt)., then we have the following estimation:
(a)(t, (t)(t) x (t)
R
W (f)
(196)
hence by Assumption 5.2 and Hölder's inequality, for some 2 depending on a, we have
(((a(tt 1 (tx(t,x)) S 1+
Σ
W., (
(197)
1=1
x'(antt,))) + Σ ( (B)()))
(198)
d (o"(xx(txx)) $1+ + Σ (. (...(t))++IBBA.) +IB,(t)P,
(199)
and
(op(((1))))
Σ
ay (,We:((W +
Σ b, (a) + (o())(t,z))
:==1
1=1
+ (/
(200)
1==1
(WW,, +)= lo(axtt,,))
$1
We.1 38 |Bx
35

---

for some p 2 1. Here, we use the mean value inequality 2n...2n 5 +...... 2 for 21, ., 2 0.
Then. by Cauchy's s inequality, we obtain that
d (Ax(t) - As(())
(t(t) - Ax(0)) = dt
dt
No
N2 (X3) ( f(X;))
2
1
(a((t, (Xi) +
(o Xz))
m
Nu
N₂
N
N N
2
(a(att, x.))12
(o())) (x1))
m
N₂
(201)
N
2
Lo(a(l, (3)11
Lūo |(x))
N₂
2
14
(t)( + (B)())) + +1 Σ
(k, 3p + Ba(tt)())
. v((
2
1
W1 + (B(()))0)) ((aoco))
m
Likewise, we also have
d (Wa((t) Wer(o)|
2 vm
1 Jm [1+1A10)1
+ Σ
(e)(t)) + (Ba(O)))))
V
dt
1=1
(202)
d [t)()) -Ba(O]] 2 mm
1 [+|AN(()]*
(t.((t) (Ba())))))
-VE(ame)) (203)
dt
Thus, if we define
Fi(() (t) Ax(0))+ + Σ Wk,((t) (0. (bx Bx(O)),
(204)
which is a Lipschitz function, then
(Ax(1) Σ (e) + (B&()) & F + +|4x(0))* + Σ W2, + 10)
(205)
Conditioning on the event B which defined in Lemma we have
Ax(0))
(o) (0) & (logm)22
(206)
hence
d (t)
(logm) 22p
(1 (1+ (i(
dt S
(207)
vm
where C' depends only on d, Ωt L. a and L()). Since L(ū(()) is decreasing along the gradient flow, then
m) 2p (1 F2()4t) v (Mmoro)) S
(log (logm)²r m)
(1) <
(1 + (2(())p).
(208)
dt Therefore, conditioning B, for sufficiently great, we have
Vm
Vm
F(t) = F(t) F(O) T(log m)
(209)
Vm
fort E [0,
36

---

Lemma F.12. Conditioning on the event B, we have
(log m) 221 vm
sup a.yen, (E (o (eco) w) =0
(210)
Proof. By definition, have
(x, =1+ (m,(, 2,,1,
(211)
where
2m
Gm (t,
o (ax((t,)) o(ax(t,)))
(212)
2m
k=1
2m
Hm (t,
A ((.n)) d'un(tt, v))-
(213)
2m
k=1
Denote (t( = ( (f, mc (tx v)), (t, x) event B, we have
Wa x + Bx. Conditioning on the (a)(0,z)] $ Vlogm
(214)
and
(log m)) (215)
(ag(t, (0, = (t) (o) x + (t) Bx(()) x
/m
then by mean value theorem,
[t((t) Ex() = (t) -o(a
= (9)) lay (1, - ax (0,z))
(216)
(log (logm)2p
E (1 + (P)
mm
for some value Ç between ag (t, and (0, 2). Then Ç O (Volomm) and
(t) (0) S (1 - (Vlog pP) (log m) 2p
(217)
/m
The same estimation also holds for (m=(()) - m7 (0) Thus,
(()))))(()) - Ex (0)
< (t) - (0)l Im (1) - (00( + |Ex (6) - EA (0) Im (0) + (0) (mx(t) - m/ (0)
(1 + (Vlogm)) )) (log (logm)2 22
2(1 + (v log m)) (log m)2 2P
log m
(218)
Vm
m
(log m)
vm
In other words.
(log m)) ())
(a(a((tz)) o(au(t a(ax(0,x)) o(an(0, y)) S
(219)
Vm
Recall that
Gm (1,,,,) 2m
Σ k=1
a (an((1,)) ((nn(())))
(220)
then
(log m) Tu
Gm (t,(t,y))) - Gm (0, x,1)
(221)
37

---

Similarly with 219 we have
(log m)
1 (o(((((,)))) (((()))))))(((( d'(ax(0,)))1
(222)
vm
hence
Hm (1, x,,) (0,
2m
As(t)] (z, F) (ax(t, d'(ax(t,))) o(ax(0,z)) d(ax(0,)))
2m 1=1
2m
(log m) -
(223)
M2 + dR²)
2m
mm
kmm
(log m)
Spa3
Vm
Combining (221) and 223
conclude that
3pot (log m
sup n.yeft tE(o,r]
Koot) (x, (2, v)1 = =0
Vm
x
(224)
Lemma F.13, Conditioning B, we have
3eps (log (logm) m) m
sup v,ye0, (E(0,2]
Kock (x, L. (o10) =
(225)
(log m)) vmn
8p48
sup e.
Koc (oco =
(226)
(log m) m
5p47
sup 2.y00, (E|U,2]
(x, C,Cy (),
(227)
Proof. For brevity, only provide the proof for Ko() here. are exactly the same.
The proof for Cy and
Note that
C2 Ke + (1,, + Hm,1,
(228)
where
Gm(t,x,y)
C,o(an(t, o(
(229)
2m
2m
Hm x,
C. (,) o (ax(t.)))
(230)
2m
k=1
By computation, we have
Σ an agW.,(tW.,,(o""((t...)) Σ b Wv (o
b==1
(231)
where b, + Σi 0,a.y. Similarly with 219 for any x, E Ω. [0, T). conditioning on B,
we have
S (log m) 2
8pt1
- (
(232)
Vm
Spt1 (log m)
(t)()x(,,))) (tan v)) - )" o(an(0)
(233)
m
38

---

hence
asy hwt.:///wwi...ccooooo///////oaad
+ -ttp://hhllllooooooooooooo////l-
(234)
+ l (ena(( - (1)
Ilg (log (logm) m) s
(log m)
S
2
(log m) _
log
vm
=
(log m) 6p13
v m
Thus,
Gm Gm,,0,,
2m
(oo(an((, o(uu(1,))) - (0, oaan (0,
2m
(235)
(log m)
/m
Likewise, we have
(,, (0, 9 (log m) Syyl
(236)
mm
In conclusion, we have
(log (logm) m) sat0
sup (a Kx (2. =0 x.yefl, (o
(237)
m
Now begin to prove Theorem 5.4
Proof. (of Theorem 5.4). By Lemma F9 and Lemma F.12
have
(v
log m
(log m)) m
Sett
sup re
[Kae (2. v) K(x, v) =
(238)
with probability Omm-3)mmm) $). By Lemma E.10 and Lemma F.13 we have
L. (oto L,K(x,v) = (v log m
Sp (log m) Vm
2
sup z,jeñ,, (E(o,T]
(239)
L (eco((,))) (, 0 (v log (logm) m
sup rE(O.T)
(240)
m
mm
L C. Kote L.LyK(n, f) =) (v
(1og log m
(Log m)
sup z,yEñ, (0
+
(241)
m
v
with probability 1 0b)
39

---

F.3 Proofs for deep neural networks
In this section, we prove Theorem 5.4 for deep neural networks, while may omit some tedious computations in the proof.
In this section, the activation function of the network to be a tanh, the depth of the network is L 2 9, and the domain Q is assumed to be convex.
It is easy to see that the NNK of üo (x:) coincides with that of tily (a:) defined by
_ww W (z)()() +b, dl(x)) (x), Vm
z xx W b0,
(x)
(242)
where me d, M...... m, ML+1 1 the widths of the layers of d/ and W E Rm b' Rm are initialized as independent standard Gaussian random variable. We also recall that
S m.. M, S Cm
for some < C.
The neural network kernel (NNK) the I-th layer 2 defined by
K0 (z, v) Vo, (x) 2o)
(243)
where 8r denotes the parameters of the first layers. As an analogue of 146), the NNK of (x+((x) is computed by
kd+1
(x, = 5ij 8,83
(((())) +
Σ W
W Wi o'
(=)( K2 (x, y).
jq
p.q=1
(244)
Our proof for the convergence of NNK to NTK initial time (Lemmal F.14 Theorem F.15 follows the discussions in Section 3.1 [17].
Lemma F.14. As m - 0o, the 1-th layer 2 (z) converges weakly in C2 g' (x), where g'(x) is Gaussian process with i...d. rows and covariance function G given by
Gx. (x. y) = =1 (z. v), G(+1(x,1) = 1+ (11 --
(245)
Proof. The smoothness of and g' arises from the smoothness of = tanh. Our proof is by induction. The proof for 1 = 1 is trivial, since
Cov(2/ (x), 2 (w)) Cov 80, w b9) 1 (I,1), i= 1....,, m1
(246)
and 2 (z), 2 (z) are independent for # j.
Assume that the statement of this lemma holds for 1,... 1- For 1 2, is well-known (see
Proposition 2.1 of (20 for example) that for every fixed x Ωt 2xx)) converges weakly to (x)x. By Lemma B. of 17), in order to complete the proof, suffices to show that for any E (0, 1), there exists some constant C₁ depending 1, 6, a, my and Ω such that
|| (z'(z)l I Ct.
(247)
Define (²(x)=
S.a a (Wh-1 b- and (x())) -lll r b--1, Then
(xx) の5' o . w (x).
(248)
By Lemma B.4 of [17, there exists constant C₁ depending only on Ω. a and 5 such that
P(A)) 1'
where A, = (1l)t())lcu SCI.
(249)
40

---

Then, by Lemma B. and 2.1 of [17], and using the fact that W and W are independent, we obtain that there exists some constant C2 depending only on f2 a and á such that
P( o 4p llaaa 2 C2) 2 P(Il llllc S A1) E( P(llp² v' S C2)A1))
P(A1) 6 7)
(250)
2 2 76.
Then, repeating this procedure, we can obtain the conclusion.
Theorem F.15. For any 1 l,.... L 1 and any multi-indices a and with |al, [8| S 2, with m fixed, as m + we have
V Ko. (x,
V
(251)
in CO2 where the neural tangent kernel K' K'(x,, y) is determined by
K (x, = (x, y);
(252)
K (x, () =1 G (z,v) K (2, g) (2,
and
G (x,)) EE (f())) (f(y))
(253)
Proof. The smoothness of Ky.sj and K arises from the smoothness of tanh.
The proof is by induction. For 1 = the proof is trivial:
Kg v) = dip (1 + (z, v\) = dzg K (xx (x,z). y). Assume that the statement is true for 1, 1.....I- | well-known (we refer to [ 43 for details):
(254)
1. For 2 2, the following pointwise convergence is
K2 ( v) 3 (., x E..
(255)
In order to extend this pointwise convergence to the uniform convergence of kernel functions and their derivatives, by Lemma B.2 [7], suffices verify that for any 8 E (0, 1), there exists some constant Ct depending only on I, 5 and Ω such that
P P[] (z, v)ll Ci) 1 8-
(256)
where ((2.),(2..)) x ? denotes the set of all the functions k(r, y). x; y E such that k(z, 3) is C for with y fixed, and is also C for y with xr fixed.
By basic inequality 2ab S a² b2 and Proposition B.7 of for any multi indices a, with Jal, |3| < 3, we have
E sup D ( L.ven
<C<
(257)
for some constant C' depending only Ω and where g' is the Gaussian process defined in Lemma F.14 Then, by Lemma F.14 and the induction hypothesis, we obtain that for any M > 0, for
any my fixed and cm < m....... mi-1 Cm, -> oc, have
8G my
lim E
sup
D DeD
AM
z.yEn
Rmm
lim E
sup
D De
M
m=0000
z.eeñ my
(258)
ma
(f AM
sup
2.gEf mi
<
sup
(o <C<00,
my
41

---

hence, if m is sufficiently great, then there exists constant C' > 0 depending only on and f such that
8ij
my
sup
D D (z
AM)
a.yE
m!
(259)
C"
M
E
sup p.yeSt
D (
A $ M
Thus, we can choose M sufficiently great (depending on and (*) such that
mi
P
sup
D
((
M 8.
(260)
z.yEf
m
Next, for the estimation of the remaining terms of Kg (z, y) in 244 we make the following decomposition:
w wi iq
(2) (1,
W wt
(z)
(261)
jp
71
W o (z) (z p (r,
mi
Similarly with 260
we can also prove that for m sufficiently great, we have
IP
w! Wt jp
(z)
(2, (x, y)
M
<.
(262)
my
(3.3(1xx1))
For the second term the right-hand side of 261 first note that there exists constant C" depending only on and Ω such that
C"
sup z.enn mi
(
M
S
(263)
M
for any M > 0 (the proof is similar with 260 In other words,
C"
F(I () > M) < M
(264)
where
F(x, u) 52 0 1m((m--)) (x o (=)
(265)
my
We also note that, by the tail estimation for Gaussian distributions, the map
pmi(m 11 + R, p(x) = Σ w W
(266)
ppg
satisfies
P(J C'log N) N- <~>0
(267)
for some universal constant C O. Thus, by selecting appropriate M and N, we can prove that there exists some constant C depending only on 8, Ω and such that
PP[| F(x, v) Illlann C"") S8.
(268)
in other words,
mi
IP
W! Wi (2(()) (I. ip jq
M <
(269)
mi
p-g=1
(3,((f(0))
42

---

Combining (260 269 and 262 together, we obtain (56). hence complete the proof of this theorem.
In the following results (Lemma F.16 Theorem F.17 and Theorem 18 we establish convergence
of NNK during training. Since the network un (z)
v2 (2)
+ (x)) consists of two
independent parts o and 2 we only need to prove for the two parts separately. Therefore, for brevity of computations, prove the results for the fully-connected network tig (z) defined in (242)
instead, and the proofs for üg are almost the same.
Lemma F.16. Define B' be the event that all the parameters 0% Wi or by are bounded by M cv log m at the initial time 0:
B' (1 S M).
(270)
then for some constant C > O, we have P(B) 2 1 Cm
Proof. The proof is similar with that of Lemma
E55
Theorem E.77. Conditioning on the event B,, along the training procedure, for any T > 0 fixed the
parameters Og Wis or by of the network satisfies
log
sup sup (1) 0x((0) O(V & (E|O,T]
(271)
m
Proof.
This Lemma is an analogue of Lemma
F.11
Recall that the evolution equation of 0g
d 02
2 No
(x,)
N,
dLz (L2 (V))+
L+1 (x:)
(272)
di
Nu
00,
N
00k
1=1
For any n = l....,d. d, by direct computations we have
1+1 (x)
........ Whe diag(ō(>"(x))) W ('.dia((((=(())) w
(273)
where W .n
is the n-th column of 1 Next, we compute the derivatives of ar. 2 1+ + (x) with respect
to W Note that
0 0
[1)
(x)
mm m. (x)( WL- W diag (2 (x))) W (274)
1
OWL
1k
Define the matrix norm || by ||M]| Eis [Mil for M (Myy). It is well-known that
[|M]]] s [[M]]- |[.]]. Also note that a = tanh has bounded derivative à S C. Then we have
a
l k. ||
I
Fzn r+ (w))
aW
I&
m4 m.
|
(275)
Mi ML
C
Σ
W
mmy ... .
L55
Likewise, for other parameters WI,, O...., N.....,..... can also prove that (we omit the tedious computations here)
i
0
244)(1))
C
(1+
Σ
(276)
awL ij
Oxn
m1 ......m... mL
82
21)((s))
C
<
(1+ Σ
(277)
OW! ig
0xm
/ml m-m
1,
43

---

for some depending only on L. Combining (276 (77 with 272 we obtain that
W - (0)
01
(278)
C(mr, + m.m.s + +.... + mmm0 + +mnd)
(1 (1+) Σ
mm ... ML
Recall that L and S m...... my Cm, hence
i
W
(1)
W (0)
S
C
(1+ Σ
(279)
at
m5/2
L.J
1,8,3
We also note that conditioning on B", we have
Σ
< Cm² Vlogm. m.
(280)
Finally, following the same procedure from that
204 to 208
in the proof of Lemma
E.1
prove
log m
F(I) S T
(281)
Vm
for m sufficiently great, where
F(I) Σ (1) - ()). 1,
(282)
This completes the proof.
Since mL+1 = 1, we denote Ko(x, y) = k-t01 (a, y) for simplicity.
Theorem F.18. Conditioning on B', for l..... L - 1, we have (K(c)(e, y) - Kotoy (2. v)1 - 0
sup
(283)
(...
as m + 00.
Proof. We start by estimating the perturbation of 2)). Note that for / 1, we have
2! =((t),, x) 2'(0(0), x) = (t) (0) + (b)(t) 8 (0))
(284)
The result 271 in the proof of Lemma F.17 in fact implies that [ ( < log and
(L) - 8a (1) x lo where || ||2 is the matrix 2-norm and | is the Euclidean norm. Thus, we obtain that
|2' (0(t), x)(0) log m
sup
(285)
x
Then, for / 2, we note that
1 (I) (10))))(00)))
=²((((), =0(00),
Vm
W (0) (0)(=(= (00)))))
(286)
Vm
(6' b')))
Again, by 271 b. we have (1) - 4 (0) S
log and (b)()) (0))) < log
Using the fact
that |a'| bounded, we also have
log m
(287)
m
44

---

We also note that
mm W (0) 12 S log conditioning on B'. Therefore, we obtain that
sup (o (0(t), - azz' (0(0), 2)1 (log m)²
(288)
m
Repeating this procedure, we can prove that for any = 1,..., L, we have
(logm)'
sup lo (((=))0))))))(00))))))))))))))
(289)
m
Next, estimate the perturbation of Doz layers.
A= 38
Denote 0f be the parameters of the first /
For 1 = 1, since (0) is linear with respect to 0, we have
2)((),(()) Da (0(0), 2)112 0, i= 1, 1. mi
(290)
Moreover, is easy see that
1 50<
(291)
where C' > 0 depends only on L.
Then, for / 2, note that
0z7
(2)
(292)
OW
mm1
DøL 22 (0,z))
w! DoLo (0,x)).
(293)
/m
Thus, using 292 291 and the fact that ب W! 12 S log m, easy to prove that (with some
tedious computations omitted) conditioning on BY, we have
log m
((()))))(()) Dgz (0(0), 1)
(294)
log m. Repeating this procedure for I = 1,... I. conclude that L++ (8(0). (Do2)+(((00).000) x1 z (log m)) - Dgz - ( 23 (log m) m
(295)
(296)
(297)
where or >> 0 is constant depending only on L.
Finally, since
Ko y) 10==++1(000 x) Doz D (0, x));
(298)
by 296
297
and Cauchy-Schward inequality, we conclude that conditioning on B', we have
[K, 3) Ko(o v) S (logm) m
sup 2.
(299)
Similarly with Theorem F.18
we also have
Theorem F.19, Conditioning on B¹, forl I,.., -- - 1, we have
sup
(2 v) C, (2., v)1 + O,
(300)
(...
sup
(2,Ko0) v) - K, (I, w)1 + 0,
(301)
(0.T)
sup [.
Kow v) C.C. Ke:0) (2. v)1 - O.
(302)
00.
45

---

Proof. The proof is similar with that of Theorem F.18
Combining Theorem F.15 F.18 and F.19 together, complete the proof for Theorem 5.4 (II. We
note that, as discussed earlier in this section, although Theorem F.15 F.18 and F.19 are proved for dlg
the proofs for dgg are almost the same. This is because the network dg (z)
(z) 22 ++1 (x))
consists of two independent parts 2 [++1 and L++ both of which are fully-connected networks in the same form of ig.. and we only need to prove for the two parts separately.
G Proof of Lemma 5.6
Since ej is sub-Gaussian we have the following tail estimation for Ey;
P(|ş) > r) < 2e -cf?
(303)
for some > 0. Thus, for any n. B > O, with great probability 1 - On we have
|z S log n, where Zj = =(X,) + Eg.
(304)
Consider the gradient flow ITO
N
2
(1)
Ke(t) (x)) (Xi)
di
N₂
(305)
N by Ky( (2. Yj) (Lusr) (Y₂) -Z;),
N
which both PDEs. To avoid thorny PDE problems, add the gradient flow of Lūg into consideration:
2
(
Σ
L. (x, (X,))
dt
No
i=1
(306)
N,
2
Σ
LoCyKoto Xi) ( - -Z)).
N,
We also recall the gradient flow (11
N,
N, Σ (()(x,)))(CC((())) - 2,), (307)
2
(((x))
K(x, X.) (x(x)) -
dl
=
Nu
N₂
j=l
No C,K(,, (x((.)
di Lūr(2)
2
N₂ Σ ((C, (Co,(X)) - Zz). (308)
No
Note that the gradient flows of the following 2 + No + N, functions
eoçe) (I), Cuoce) (z), (oce) (X1), doce)( (N..)) (.... (X)) Litte (YN,)
(309)
form a linear ODE system, and the gradient flow (TO) is completely determined by this system. Thus, by the uniform convergence of kernel function (Lemma F.12 and Lemma ME and the continuous
dependence theorem of ODE (Lemma G.3 we obtain
Lemma G.1. Given T > 0, N.. N2 fixed, we have
sup sup (0)()(()) - dx(x)) + 0,
(310)
Lejo,T] X.Y;;
sup sup (C)) Lū,(x) tE
(311)
in probability. The randomness arises from the network initialization.
46

---

Proof. We apply Lemma
G.3 by setting k = 2+ + No + N, and
01x2
(x,
N, (,
0 01x2
C (x,X)
Ny Kor
A(t)
N. 2
(312)
ON.22
N. (X(n)(XXX) (X,X)
N, (x,1
02
Ca Ko(t) (Y,
Ko
N.
Ni Kar ()
2
2 Ko Ko(t) -2 N
B(1) =
(313)
N7 Kou)) (X, .)
2 CoL (Y,))
N,
01x2
3. K(x,))
N, (((
01x2
N ((,,
2 -N, -.C.C,KKE,)))
A
(314)
0 N2X2
-(X,) N. K(X,
O2
N. (X,,
Nj (.C.CKK(Y.Y))
2 N ().
No Y)
B
N7 -x,,
(315)
where 0mxn denotes the m X n zero matrix, and Ko(t) (X (X, = (X,
(,Ko())(XX1) = (X,, (o,,)(X,,,,))aaal Kj))N,xN,,
(316)
L Koy = (Koco) (Y,, (j))N))),
(Y))) = (Koc) YJ))NNNN,
(317)
K(X,X) = (K)x;,
CKK(X,Y) = (K(X,,, Kj))N,xNy
(318)
(,K(Y,X) (K(Y, (1)NN,N))
ColyK(Y,Y) (K(Y, 1j))
(319)
Ke(o (x, X) (I, Xj))IXN,>
LyKotolo, (Koco) Yj))
(320)
K. (r, X) (Keto (2, Xj))axN..
(o,, Y) = (Kaco (I,, Yi))
(321)
K(x, X) = (K(x:X)))xNN, Cn = (K(Ixx)))IX,, LolgK(I,Y) (K(Y, Yj))INN
L,K(x, = =(K(x, Vi))xNN
(322)
(323)
(Zi...., ZN,
(324)
Note that
K(x, v)1, 1,, ((,,)),,,, (xx,,,)), K, w)1, l S Mr
(325)
for y E Ω2 where the bound Mr > 0 depends only on
Denote as || Iloo the maximum norm: |1 maxx, layyl for matrix (ag)). Then
|||| 00 S 2
N₂ + N₂
Mx.
(326)
By 304
also have
B S log m.
(327)
with great probability 1 - Om 0m11))
For shallow networks, by Lemma F.12 and Lemma F:13 we have
1 o
(log m) m
max ||(t)) Alloc Nu
(328)
N₂
(log m)
Sp4z
max [B(t) - Blloc
(329)
[E(0
N
m
47

---

with probability P(B) 2 1 - (mm-1/2).
Finally, by Lemma G.3
we conclude that
sup sup (x) -+(()) {Q(m), ve(o.T).meEf X,,Y,
(330)
sup sup (x) Lū(()) - Q(m).
ve(o,T),.eet X.YY,
with probability - ((m-1/2) where
(log m) 2n77
Q(m) D(T, Ny,
(331)
V
for some constant C > 0, and
27(2 N. 271--
N, + +Ny
N,
N
(332)
For deep networks, using Theorem E.15
F.18
and
F.19
we can also complete the proof similarly with
the above procedure from (328) to (332
G.1 Continuous dependence of ODE
Lemma G.2. Assume that X (x) (t)) is the solution to the following k-order ODE system:
d
dt X(t) ((t)x((t B(t),
(333)
where A(t) and B() are continuous with respect to t E [0, (0) = 0, then
max AE(O,T]
(|x( & Thexp Th max tE[0,7]
[A(()]l>)
max (E
[[)
(334)
Proof. Note that
d
K (t) < (,(00)))1 & |1((0)) Σ
[2(()] = dt
(2)(t)+ (335)
dt
Let M(I) = Eí= [ then
d di M(1) s M() +
(336)
hence by Grönwall's inequality,
|x(1) max M(t)= Tk exp (E
max
Tk max tE(O.T) [|A(())<x)
max :E(0,]]
[B(t)
(337)
rE[o.T]
Lemma G.3. Assume that X(1) = sx(()) and =(t)=(11(())) ((((),,.,,,(()))) are solutions of the k-order ODE systems
d
d ( AY + B,
X(t) A(t)x(t) B(t). dt
(338)
dt
respectively. If (0) = (0) = 0. A(I),(L) are continuous with respect to E [0, A is a fixed x le matrix, B is a constant i- k-dimensional vector and
max tE[D.T]
||((t) Al <E.
max rE
[B(() - Blloo <E
(339)
for some > 0, then
[x((t) - oo) s TK (I T THDAAlo))))) (s+ +(Al)o))
max
(340)
tE (0
48

---

Proof. Denote ((t) (t) - (t). Note that
dt Z(t - (t)) + R(t),
(341)
where
R(t) = (A(t) A)V B(t) B.
(342)
By Lemma G.2
we have
max ( S 3 TTIIA (E(O.T]
(343)
hence
max [ & (1 (1+ [1+Tk]]] T Bl rE[O,T]
(344)
Again, by Lemma G.2
have
[()))l TTe (| (1+ [ THIAAl>)
max t
(345)
H
Sphere and Torus
Now we prove Theorem 5.9
Proof. Let 0 = p < nn <... be the eigenvalues of (-A)-1, and Pi h: = 0,..., O,...,(()) be the eigenfunctions of pu:
AD. -,
(346)
Here, N (i) is the dimension of the eigenspace of p4. M St. then A; = i( 1), and ig are the spherical harmonics; If A T then i are products of trigonometric functions. Note that { forms an orthonormal basis of (S").). and if the Fourier expansion of f is
80 NO
Σ
Σ
Y
(347)
then the unique solution (55 is
00
N(
filk
up
Y
(348)
where the constant Up is selected such that u(p) = Ulp-
Note that neural tangent kernel K (ær, is inner product kernel on M: K(x, v) F( y)) for
some function F [_1,]] 1 + R, where (...) is the inner product in gd+1 if M = gd c Rd+1
or the inner product in TR 2d if M = T c 2d Then by Funk-Hecke formula (see Section 1.8 of [16] for example), the eigenspaces of K coincide with the eigenspaces of (-A)-1 and the Mercer decomposition of K becomes
2
NL
K(x = Σ A Σ
(z)
(349)
where An 2 11 ......,.. + 0 as - 00.
For / N, let
NG
fuN fu Y
U1 Up.!
(350)
P
49

---

where the constant Up. is selected such that ux(p) up. Then E H, and
N(6)
/ Asm Aupl? 2 Σ Σ
L(u) = d(u;; u²) - -1)
u- (w)1
2 +0
(351)
km1
as co. Thus,
inf d(u, u²) = 0, ue
(352)
which implies that u² H.
H.1 Proof of the negative example
Proof. Recall that u" (e = t² and Au"(c = for E (-1, 1). For any u E H. let ag and
by be the Fourier series of v:
00
o( ag Σ
(ax cos(k)t) + by sin
(353)
Since v is in the RKHS, then by Theorem 14, we can differentiate term by term:
Au(c
agk² cos(krt) - kykk sin(krt))-
(354)
The PINN distance between v and u² is
/
L(v) d(v, u u²)² (p(p) u² (p)l
IAv A
= =1u(p)1 +
aak cos(kTt) - bak² sin(krt)) - dt (355)
2
= -1((p) +
kst (ak + (2) + 4424
and the equality holds if and only if 0. Thus, 0 the unique minimizer of the loss function, and the population gradient flow is solved by o = O, E [0, oo).
H.2 An extra example
In this section, provide example where the physics-informet kernel gradient flow is consistent even though the nomohonogessous term f in the problem (25) is not bounded. It is worth noting that in this example, the convergence speed of the PINN loss L can be arbitrarily low.
Consider special case of the equation (25
Au = fr
on S1,
u(p) = 0,
(356)
where p (-1.0) ii If (0) Ex fy sin quÔ, then the solution (356) is u" (0)
-g f49
-- 2 sin 9TB.
Consider the physics-infiee informed kernel gradient flow estimator û, of an inner-produce kernel k(z, y) in the form
k(0, = ao + (a. cos q cos qrip + b, sin qro sin qTyp.
(357)
Note that the NTK S¹ will be in this form. addition, we assume that by x xq for some B > in order to ensure that An,,h(0, ip) is well-defined.
In this case, the evolution equation of ún is
d (0) -2 (Y.
(358)
d
50

---

where we set the noise Ej to be Gaussian (Ey ~ N(,, 0²) for some o² > 0), and
1(0, ()
Σ My sin sin gmop 9-1
Pq by = 7/5 7= 3 4
(359)
(We choose B sufficiently great such that k(0, yp) is tiieediifferentibbee and k0, sp) is well-defined.) This is exactly the kernel gradient flow equation for classic regression problem (see Example 2 of 45)). Along the gradient flow, (p( = 0 for any t 0, where p = hence
L(ū.) = Au l'
(360)
It is shown in Section 4 of [5 that the embedding index of 1: is 1/7. For any 0, let [7]] be the s-interpolation space of the RKHS of k. We refer to Section 2.1 of 45 for more information about RKHS, interpolation spaces and relative smoothness.
Finally, by Theorem 1 of [4],, the kernel gradient flow is consistent for any f* e [H] We note that in this circumstance, f* is not necessarily bounded.
Moreover, by Theorem 2 of [45], for any d E (0, 1). when Ny is sufficiently great, there exists function f E [7]] such that
L(ū.) 2 N मम
(361)
with probability at least 1 1- 6. Since $ > 0 can be arbitrarily small, we conclude that the convergence speed of L(as) can be arbitrarily low.
I Experimental Results
Although the results of this paper are fully theoretical, we still provide several experimental results to verify the results. Our small experiments were implemented 2023 with CPU 19-13900HL RAM of 16GB and GPU RTX-4060 Laptop 8GB). The codes were written in Python, and the Pytorch package was used. The execution time was less than an hour.
personal computer (Legion Y9000P
L.1 Experiments one-dimensional interval
In our first experiment, we focus on the 1-dimensional problem:
u" (xxx) 2, (-1,1); (u(-1) = u(1) = 0,
(362)
whose solution is
(x)= x x [-1,1
(363)
We select the training sample set be
X1 = -1
X₂ = 1;
(364)
-088,
Y/2 0066,
Ys = 0.6, Yo 0.8.
(365)
Hence the sample size is N₂, 2, N₂ 99
Then we train network (g' (a:) in the form 145 depth = 1 and width m. and activation function a = tanh to minimize the empirical loss (8). We use the discrete gradient descent to approximate the continuous gradient flow. The training time is rate is set to be 10-4, hence the number of iteration of gradient descent is 105.
be T 10, and the learning
Denote Kitt (zr, y) the NNK of an compare Kote) (2. y) with ((x, y), and compare (x();(z)) with the gradient flow (e(x) defined in (IT) on the test point sets, which is defined by
(x), and recall that K(x, y) is the NTK. Our goal is to
2test (Z = 0.9, Z2 -...... Zi8 0.8, Zig = 0.9),
(366)
23 Ztest Ztest [22, 41) i.j I,..., 19),
(367)
Trest {ti 0, to = 0.1, t3 0.2, ,110 9.9, t1o 1}.
(368)
51

---

By our numerical experiment, if we set m = Mo 10483760, then the NNK function of à Mo (a) satisfies
1
Σ
KK (2. v) K( $ 10-9.
(369)
(1.2))
Σ
1La (1) 0(1) (2, -0,,,))-L,kk (x,(x, w) $ 10-9,
(370)
Zleat
(2.0)
Σ
ICx 1
(2 v) S 10
(371)
Z
(2.9)
for any & Teasts hence it is reasonable to use ugde (x) as an approximation of (z()):
K Mo (I, s K(r,y), 0(1)
d #8(0 (x) s ûz(z).
(372)
For
m 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240.
(373)
the figure of the mean square distance between (1) (z. v). Kiy (x, v). Cx (a, and (Ko p(E) (z, u), f K Mo 8()) (2r, v), L. by K o(t) (ar, v)), that is,
t +
Σ
(0 v) (xt)) y)1
3 2lest
(2, lest
(1.K01)(E11)) L. K Ka y)1
(374)
(2.9)EZ)est
Σ
1L,LyKol (x, - C. L, (z. v)1²
3 test (2.9)
for E Trest is shown in Figure
The plot of the mean square distance between (i (z) and ab(e) (x)), that is,
( - ab (z)) tE Trest:
(375)
Ztest
is shown in Figure
The plot of the mean square distance between Lüm (xr) and Lūgl (x), that is,
(Lugt)())) Lugl (z))², E Trests
(376)
Brest
#EZicut
is shown in Figure 3
1.2 More experiments on one-dimensional interval for deep networks
Although we only provide proof of Theorem 5.4 (II) for 2 9, in this section we offer experimental results that verify 5.4 (II) for L 9.
In particular, we set L 4, and take the network to be úg defined 145) with activation function lanh, depth L = 4 and width my m2 mg 1744 m for
m 80. 160, 240, 320.
(377)
The training time is set to be T = 4, and the learning rate is set to be 10-1 hence the number of iteration of gradient descent is 10 The network still trained to solve the problem (622] the previous subsection.
52

---

m=40
0.025
m=80
m=160
m=320
0.020
m=640
m=1280
m52560
0.015
m=5120
m=10240
0.010
0.005
0.000
Figure 1: Illustration of Theorem 5.4 The evolution of the distance between Koçe) (I, 8) and Ke(x, y). See Appendix [I.]] for details.
(x-axis: training time t; y-axis: mean square distance between
(Ko L, C, K and (K, L,K, kyCy on the test point set)
m=40
0.005
m=80
m=160
m=320
0.004
m=640
m=1280
m=2560
0.003
m=5120
m=10240
0.002
0.001
0.000
10
Figure 2: Illustaaioon of Lemma 5.6 The evolution of the distance between ( (-) and a.(+). See Appendix [.] for details.
(X-axis: training time t; y-axis: mean square distance between ( (-) and (e(-) the test point set)
Denote as Kin (x. v) the NNK of (o(o)()), (2), and recall that K y) is the NTK. Our goal is to compare K (x, y) with K(x, y) on the point sets, which is defined by
Ztest {21 = 0.9, 22 -0..... Zis 0.8, Zig = 0.9),
(378)
23 Zrest Zrest {(Z,, Zz); i,j -, 19),,
(379)
Trest {ti 0, t2 0.1, t3 0.2,
t100 999, t10 1}.
(380)
53

---

m=40
0.005
m=80
m=160
m=320
m=640
m=1280
m=2560
m=5120
m=10240
0.002
0.001
0.000
2
10
Figure 3: Illustration of Lemma See Appendix for details. (x-axis: training time I; y-axis: mean square distance between Lūece) (.) and Lie(-) the test point set)
5.6
The evolution of the distance between Lūn()) (-) and (.,()).
simple numerical simulation reveals that for Mi 10240,
KK (z. v) K(x,, v)1 10-1,
(381)
Zreat
(m.J))
Σ
1ex (2. - Kx, w) $ 110
(382)
2
0(1)
(
Σ
Mi 1C 0(1) (z. v) C.c,K(2, v) 10-4,
(383)
(x.0)E
for any E Trest hence it is reasonable to use K KM (I, y) as an approximation of K(x,)) y) in the numerical simulations:
K(x,y)
M1 (z, 9).
(384)
For m
160, 240 and 320, the figure of the mean square distance between 0ke) (1, v), C. Km B(t) (x,, v), La fy Km (z. v)) and (K(x, C, K(r, K (x, v)), that is,
t -
Σ
K (z, -()))) 1)
3/ 3|2
(u.))e2?
Co (x
3/ 3/222s1
(385)
(r.y)
Σ
((,)))
3/ 2
(
for E Trest is provided in Figure
As shown in Figure as m increases, the distance between K (z. y) and K(x, (x, v) decreases approximately to zero.
54

---

0.08
m=80
m=160
m=240
m=320
0.04
0.02
0.00
0.0
0.5
3.0
3.5
4.0
Figure 4: Illustration of Theorem 5.4 The evolution of the distance between Ko(e) (I. v) and K K(x, v). See Appendix [ details. (x-axis: training time y-axis: square distance between
(Kmay B(1) La Km La Le Km)) and (K, L. K, L L2hy K) on the test point set)
B(1)>
1.3 Experiments ring
In our second experiment, we consider the following Poisson equation on St:
Au(e
p -T,
E [-1,
(386)
u(p)
=
whose solution
7
u" (e!ar)
20 5 _
r=1-111]
(387)
60
The network üg is set to be in the form (145) with depth L 1 and width 2m 3200 fixed and activation function o = tanh. The learning rate is set to be 0.01, and the training time set to be T = 1000, hence the number of iteration is 10a.
We train the network with different sample size. For sample size N, the boundary sample set contains only one element X; e- while the inside sample set contains N elements:
2j 2
Yn = exp
in
j j= 1,2 N
(388)
hence N,, = and N, N - 1. The test point set is set to be
in in(111+ 10))
Ztest
exp
(389)
For sample size N 10, 20, 40, 80, 160, the plot of the networks on test point set after training, that is,
r uo(o (e)
Ztest
(390)
is shown in Figure 5
worth noting that in this example, since tanh smooth function whose high-order derivatives are all bounded, then H is contained in (SS(S)) (see Corollary 4.36 of (36) or Theorem A.1). On the other hand, the solution u* not (S+),, hence is not contained in H. However, u² lies indeed in
the larger space H. and the model can still fit the solution well, is shown in Figure 5
55

---

sample size=160
sample size=80
0.3
sample 4ize=40
sample size=20
0.2
sample size=10
True solution
0.1
0.0
-0.1
-022
-0.
-0.4
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Figure 5: Illustration of Theorem 5.9
The result of PINN trained for Poisson equation problem.
(X-axis: r [-1, y-axis: the PINN estimator ( (ctrr).)
It is revealed in Figure 5 that the sufficiently wide PINN estimator approximate the true solution u* E H. even if ut" does not lie in the RKHS of the NTK.
The third experiment is the same with the second experiment, except that substitute f(r) = ق with g(r) = in the problem setting (386). This is a numerical simulation of the negative example discussed in Subsection 5.1. The result shown in Figure
-1
sample size=160
-2
sample size=80
sample size=40
sample size=20
sample size=10
-3
True solution
-4
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Figure 6: Illustration of Proposition] 29
The result of PINN trained for the negative example. As shown in the figure, the estimator converges to 0 as sample size increases, hence keeps distant away from the true solution. See Appendix I.3 for details. (x-axis: [-1, 1]: y-axis: uo(r) cir
Figure 6 reveals that when the network width is sufficiently great (2m = 3200), as the sample size increases, the PINN estimator gradually converges and fails approximate the true solution successfully.
56

---

1.4
Extra experiments on multi-dimensional torus
In this subsection, we provide extra experiments on multi-dimensional torus that illustrate Theorem
5.9
Let T C I be the three- direeddimensional torus:
T = {(ElTO)
iTo, , ei=l)): 0;
0, E (-1,,),, i == 2,
(391)
Consider the following Poisson equation on To:
Au= =
on T
(392)
((()) = 0 at = (e it in , (
where
3
1(01,02,00 = Σ (0 - -²))
(393)
i=1
The solution to this problem is
1
7 60
(01, 02 0z) = Σ
0
0 +
0;
(394)
20
6
The network úg set to be in the form (155) with depth L, = 1 and width 2m 3200 fixed, and the activation function is a = tanh. Note that tanh is smooth, then the NTK function is also smooth. Since the solution u² not smooth, then u does not lie in the RKHS of NTK by Theorem A.I
Then we train the network úø to learn the solution u². We repeat this experiment for 50 times. In each time, we draw 1000 samples from the uniform distribution of T the learning rate is set to be 0.01, and the training time is set to be 100, hence the total number of gradient descent iteration is 10
Finally, we present the relationship between the training time / and the average PINN loss L(ūgrs) of the 50 experiments, as well as the region within two standard deviations, shown in Figure 7
0.6
0.4
PINN
0.3
0.2
20
40
60
80
100
Figure 7: Illustration of Theorem
5.9
The result of PINN trained for the problem (392
(x-axis: training time [0, 100]; y-axis: the PINN loss L(ūg())
It is revealed in Figure 7 that PINN can estimate the solution u² to the problem well, as long as the network width and the sample size are sufficiently great, even though the solution v" does not lie in the RKHS of the NTK.
57


Reviews:
[{"review_id": "ogr68SOkI2", "reviewer": "Reviewer_VWqv", "summary": "This paper investigates the consistency of Physics-Informed Neural Networks (PINNs) applied to second-order elliptic partial differential equations (PDEs) with Dirichlet boundary conditions. The authors provide a rigorous theoretical framework built upon the Neural Tangent Kernel (NTK) theory to analyze the training dynamics and convergence of PINNs. The primary contributions of this paper include:\n\n1. The derivation of necessary and sufficient conditions for the consistency of PINNs under the assumption that the PDE solution lies in a certain Hilbert space $\\overline{H}$.\n\n2. A non-asymptotic bound on the loss of PINN estimators under stronger assumptions (i.e., the solution lies in the RKHS of the NTK kernel).\n\n3. The construction of a counter example where the PINN method fails to achieve consistency.\n\n\nWhile the theoretical results are correct and address important aspects of PINNs, they are limited to the analysis of single-hidden-layer networks. Extending these results to deep neural networks (DNNs) is critical for practical relevance, as DNNs are the dominant architecture in real-world PINN applications.", "questions": "The analysis is for two-layer NNs, which are rarely used in practice. Please consider the analysis on deep neural networks.", "limitations": "The results rely on the assumption that the PDE solution lies in a specific Hilbert space $\\overline{H}$ or RKHS, is it possible to extend the results in Sobolev spaces?", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a mathematically rigorous framework for analyzing the consistency of PINNs.\n\n2. The use of NTK theory to connect the training dynamics of wide neural networks with kernel methods is a theoretical contribution.\n\n3. The construction of a counter example for the potential failure modes of PINNs and emphasizes the importance of theoretical analysis before applying PINNs to real-world problems.\n\n\nWeaknesses：\n1. Limited Practical Relevance. The analysis is restricted to single-hidden-layer networks, which are rarely used in practice. Modern PINN applications predominantly rely on deep neural networks (DNNs), and the results presented here may not directly extend to such architectures.\n\n2. Strong Assumptions. The results rely on the assumption that the PDE solution lies in a specific Hilbert space $\\overline{H}$ or RKHS, which may not hold in many practical scenarios. These assumptions should be carefully validated or relaxed for broader applicability.", "was_revised": true, "final_justification": "I would like to keep the score.", "rebuttal": "We sincerely thank Reviewer VWqv for your feedback. We greatly appreciate the time and effort you have devoted to reviewing our paper. In response to your questions and concerns, we would like to offer the following clarifications and explanations:\n\n1. Concerning multi-layer networks: \n\nWe thank the reviewer for the valuable advice regarding the importance of analyzing multi-layer networks, and we fully share your concerns on this point. We would like to take this opportunity to offer some clarifications on this topic: \n\nThe two-layer networks was adopted in this paper only for the purpose of brevity of computations. Actually, our results hold for multi-layer networks. We will add a proof for the multi-layer networks in the revision. For your convenience, we provide a sketch of the key steps below:\n\n(I) First, the two-layer or multi-layer structure of the network is required only in the proof of Theorem 5.4 in Section G, which establishes the convergence of NNK to NTK (The NNK and NTK are defined in Section 3.1). Our discussion of multi-layer networks will focus primarily on establishing the multi-layer counterparts of the results presented in Section G.\n\n(II) Section G begins by establishing the convergence of NNK to NTK at initial time. Following the approach taken for the two-layer case, we will first derive an explicit expression for NNK and NTK of multi-layer network that parallels the results in (162)–(163); This representation will then help us prove the desired uniform convergence at initialization.\n\n(III) The remainder of Section G turns to the convergence of NNK to NTK during training. Once again guided by the two-layer argument, we establish—via an analogue of Lemma G.11—an estimation for the evolution of the multi-layer network parameters, which in turn helps us establish uniform convergence of during training.\n\nThank you for raising this point. We will incorporate the related discussions on multi-layer networks into the revision. \n\n\n2. Concerning the RKHS assumption: \n\nWe thank the reviewer for raising the concern regarding the assumption that the PDE solution lies in the RKHS or in $\\bar{\\mathcal{H}}$, which may appear strong and difficult to verify in general. We would like to take this opportunity to clarify this point and address any possible misunderstandings.\n\n2.1 First, we would like to clarify that Theorem 3.12 is the only result in the paper that requires the PDE solution to lie in the RKHS. This result serves merely as a supplementary result. We include it in this paper because it arises as a natural and interesting corollary of the proof of the main theorem. \n\nTo better reflect its auxiliary nature, we will relabel it as Corollary 3.12 in the revised version of the manuscript.\n\n\n2.2. Second, the condition that the PDE solution lies in $\\bar{\\mathcal{H}}$ plays a fundamental and, in fact, indispensable role in our analysis of PINN. \n\nOn the one hand, Theorem 3.7 indicates that if the true solution lies outside $\\bar{\\mathcal{H}}$, then the PINN estimator cannot converge to it. Consequently, deriving a consistency result relying on the assumption that the PDE solution lies in Sobolev spaces is equivalent to identifying conditions under which a given Sobolev function belong to $\\bar{\\mathcal{H}}$. \n\nOn the other hand, we believe that the space $\\bar{\\mathcal{H}}$ could be quite large in practice. In particular, when $\\mathcal H=H^\\beta=W^{\\beta,2}$ or $\\mathcal{H}$ is the RKHS of the NTK on sphere or torus (cases that might be of your interest), we have shown that $\\bar{\\mathcal{H}}$ coincides with the Sobolev space $H^2=W^{2,2}$. \n\nWe hope these clarifications help address your concerns. Thank you once again for taking the time to read our response and for your thoughtful feedback."}, {"review_id": "HVt6ziqX6K", "reviewer": "Reviewer_yJxF", "summary": "The paper investigates the consistency (convergence property) requirement of PINN when applying to second order elliptic PDEs. Notably $f$ is not required to be in $\\mathcal{H}$, but rather in the closure of $\\mathcal{H}$ in $H_2(\\Omega)$, which is also the necessary and sufficient condition for consistency. Based on the condition, the authors provide a notable pathological example where $f \\notin \\mathcal{H}$, which makes PINN inconsistent. It is able to deliver a non-asymptotic convergence bound if $f \\in \\mathcal{H}$.", "questions": "- In line 47, it is mentioned that \"we provide a different perspective, a new mathematics framework known as NTK ...\", actually using NTK to understand theoretical properties of PINN is not new and has been conducted in many works.\n- It is mentioned that $f$ does not need to be in $\\mathcal{H}$, but only needs to be in its closure w.r.t. $\\mathcal{H}^2(\\Omega)$, known as $\\overline{\\mathcal{H}}$, so what are the elements that are in $\\overline{\\mathcal{H}}$, but not in $\\mathcal{H}$? Are there more negative examples can be introduced in section 5.2?\n- For the positive example, the Poisson equation is provided which satisfies the codiagonalization assumption, does the theoretical analysis in this paper requires this assumption too?\n- Is there any intuition why the theoretical analysis can get rid of the standard capacity and source condition? What aspects make it distinguish from previous work? \n- Is there a regime where the equation is \"consistent\" to solve, but the convergence speed is very low which makes it unsolvable in practice?\n- What happens if we conduct analysis on first-order or higher-order PDEs? Some discussions on the necessity of conducting analysis on second-order elliptic PDEs would be beneficial. Can we bring the similar insights to other PDE problems?", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors introduce notion of consistency to area of PINN, which is a weaker notion compared to previous works which establishes non-asymptotic bounds. They also derive necessary and sufficient condition, which is novel and interesting.\n- The proofs do not require on standard capacity and source condition, which is standard in previous literature. This is pretty novel in theoretical sense.\n- The negative example is quite intriguing and help understanding the properties of PINN.\n\nWeaknesses:\n- The analysis is very restrictive since it is only conducted on the second order elliptic PDE.", "was_revised": true, "final_justification": "The authors have addressed my concerns and I would like to keep my positive score.", "rebuttal": "We sincerely thank Reviewer yJxF for your feedback. We greatly appreciate the time and effort you have devoted to reviewing our paper. In response to your questions and concerns, we would like to offer the following clarifications and explanations:\n\n~\n\n1. Concerning “a different perspective...”: \n\nWe appreciate the reviewer’s attention to this point and apologize for the ambiguity in line 47. Please let us clarify some possible mis-communications. \n\n1.1. Our original intent was to highlight the distinctions between our work and the previously mentioned references in line 38--45 that focus primarily on analyzing empirical risk minimizers without considering network training dynamics. \n\nFor example, [16] establishes convergence rates for statistical and approximation errors of the empirical minimizer of PINN loss for second-order elliptic equations; [27] studies the convergence of the empirical minimizer of the PINN loss with H\\\"older regularization term for both second-order elliptic PDEs and second-order parabolic PDEs; [8] analyzes the convergence of the empirical minimizer of the PINN loss with Sobolev regularization term for a general linear PDEs. This sequence of works concentrates on convergence analysis of empirical minimizers of the PINN loss (with or without regularization). \n\nIn the contrast, compared with these works, our work adopts a different direction by employing NTK theory to incorporate the dynamics of network training into the convergence analysis of wide PINNs. We sincerely hope that this perspective may offer a constructive complement to the existing literature on PINN convergence. \n\n1.2. We fully agree with the reviewer that there are many related works using NTK to analyze PINN. \n\nFor example, [1] investigates the spectral bias of PINN by examining the spectrum of the NTK of two-layer PINN for linear PDE problems with boundary condition (The term \"spectral bias\" refers to a well-known pathology that prevents deep fully-connected networks from learning high-frequency functions, as discussed in section 5 of [1]); [2] also studies the spectral bias of PINN, while the discussions in [2] were based on analyzing the NTK of multi-layer PINNs for linear PDE problems without boundary condition, concluding that the differential operator in the loss function of PINN does not intensify spectral bias if the boundary condition is absent; [3] extends NTK analysis to PINNs for non-linear PDEs, and suggests that the second-order optimization methods may alleviate spectral bias in training PINNs for non-linear problems. [3] also uses the NTK theory to establish convergence guarantees for the empirical loss of two-layer PINNs under specific assumptions on the network training. \n\nWe notice that a relatively small number of the existing NTK-related studies have directly provided consistency results for PINNs. This is where we hope our work could make a meaningful contribution. \n\n1.3. We appreciate the reviewer’s suggestion and will revise the paper accordingly. Specifically, we will remove the sentence in question, add more related works on NTK and PINN into the references, and include a table comparison of these references. \n\n[1] Wang S, Yu X, Perdikaris P. When and why PINNs fail to train: A neural tangent kernel perspective[J]. Journal of Computational Physics, 2022, 449: 110768. \n\n[2] Gan W, Li Y, Lin Q, et al. Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators[J]. arXiv preprint arXiv:2503.11029, 2025.\n\n[3] Bonfanti A, Bruno G, Cipriani C. The challenges of the nonlinear regime for physics-informed neural networks[J]. Advances in Neural Information Processing Systems, 2024, 37: 41852-41881. \n\n2. Concerning elements in $\\bar{\\mathcal{H}}$ and negative examples: \n\nWe thank the reviewer for raising this question. Please allow us to clarify some key points regarding this topic: \n\n2.1. We fully agree that the characterization of $\\bar{\\mathcal{H}}$ is crucial in the analysis of PINN. In this paper, we examine several situations in this paper where the elements of $\\bar{\\mathcal{H}}$ can be completely characterized. \n\nThese situations, which might be of your interest, include: 1. The Sobolev RKHS $\\mathcal{H}=H^\\beta$, whose closure $\\bar{\\mathcal{H}}$ coincides with $H^2$ if $\\beta>2$; 2. The RKHS of the NTK on spheres or tori, whose closure $\\bar{\\mathcal{H}}$ is shown to be the Sobolev space $H^2$, as is shown in Theorem 5.9. \n\n2.2. In addition to the example presented in Section 5.2, it is also possible to construct further negative examples. For your convenience, we provide a brief sketch of such a construction below. \n\nConsider the Poisson equation $\\Delta u=f$ on the $d$-dimensional cube $[-\\pi/2,\\pi/2]^d$, where $f=\\cos x_1+\\dots+\\cos x_d$. Its solution is $u^* =-f$. If the kernel function $k$ is set to be a translation-invariant kernel ($k(x+\\pi m,y+\\pi n)=k(x,y)$ for any $m,n\\in Z^d$) such as the NTK on torus, then its eigenfunctions are Fourier basis functions, and we claim that the integral of the laplacian of any function in $\\bar{\\mathcal{H}}$ must be zero (the proof is similar with the discussions in section I), hence the solution $u^*$ does not lie in $\\bar{\\mathcal{H}}$. \n\n3. Concerning the codiagonalization assumption:\n\nWe would like to clarify that this assumption is not required in the main results of this paper, including Theorem 3.7, its application to network (Theorem 5.8) and the convergence of NTK (Theorem 5.4). In fact, it is used only in the construction of the examples in Section 5. \n\n4. Concerning the capacity-source condition: \n\nWe thank the reviewer for raising up this question. We would like to clarify several reasons why the capacity-source condition is not adopted in this paper: \n\n4.1. The capacity condition states that the eigenvalues $\\lambda_i$ of the kernel function satisfies polynomial decay rate $\\lambda_i\\asymp i^{-\\beta}$ ($\\beta>1$), while it is not the property that our analysis exploits. Instead, our discussions on PINN rely on the differentiability of the kernel function, hence we simply impose the $C^2$ assumption (Assumption 3.4 and 5.2) directly.\n\n4.2. The source condition, which states that the target function lies in the $s$-interpolation space the RKHS of the kernel function, is used to evaluate the generalization ability of kernel methods in related literatures. For example, in previous works such as [4,5], the source condition helps determine the generalization error convergence rate of kernel ridge regression. In contrast, our current work focuses primarily on consistency rather than the convergence rate of the generalization error. Moreover, the source condition in the context of PINNs has not yet been well described. We plan to explore this topic further, and aim to derive convergence rates for the generalization error of PINNs in future research.\n\nWe hope these explanations are helpful in addressing your concerns on this topic. \n\n[4] Fischer S, Steinwart I. Sobolev norm learning rates for regularized least-squares algorithms[J]. Journal of Machine Learning Research, 2020, 21(205): 1-38.\n\n[5] Zhang H, Li Y, Lu W, et al. On the optimality of misspecified kernel ridge regression[C]//International Conference on Machine Learning. PMLR, 2023: 41331-41353.\n\n5. Concerning low convergence rate: \n\nWe thank the reviewer for this insightful question. \n\nWe have successfully constructed such an example where PINN is consistent while the convergence can be arbitrarily low. We will include it in the revision. For your convenience, we provide an outline of the construction below: \n\nConsider the Poisson equation (24) on the one-dimensional ring $S^1$. We take the non-homogeneous term $f$ to be $f(\\theta)=\\sum_{i=1}^\\infty f_i \\sin i\\theta$. Let $k(\\theta,\\varphi)=a_0+\\sum_{i=1}^\\infty(a_i\\cos i\\theta\\cos i\\varphi+a_i\\sin i\\theta\\sin i\\varphi)$ be a Mercer kernel on $S^1$, and assume that $a_i\\asymp i^{-\\beta}$ (Note that the NTK on $S^1$ will be in this form). Let $\\hat{u}_t$ be the kernel gradient flow estimator defined in (12). Then we claim that, for any $s>1/\\beta$, there exists some $f_i$ such that $f_i/(b_i^{s/2} )$ is an $l^2$ and the convergence rate of $L(\\hat{u}_T)$ is no greater than $n^{-\\frac{s\\beta-1}{2s\\beta}}$ for any possibly selected stopping time $T=T(n)$ ($n$ is the sample size). Since $s$ can be arbitrarily close to $1/\\beta$, then we conclude that the convergence rate $n^{-\\frac{s\\beta-1}{2s\\beta}}$ can be arbitrarily low. A detailed proof will be provided in the revised paper. \n\n6. Concerning other types of PDE problems: \n\nWe thank the reviewer for this question and would like to provide you with some clarificatins on this topic: \n\n6.1. We are confident that the main results--including Theorems 3.7, 5.4, and 5.8--extend to a broad class of linear differential equations. The key derivations and estimates in the proofs remain valid for the general linear setting with only minor, routine adjustments.\n\n6.2. We will add discussions on what we benefit from the second-order elliptic equation into the paper. The reasons why we conduct analysis on this type of PDE include: \n\nFirst, the Green function method of this type of PDE helps us establish the norm control (Lemma 3.9), which provides an estimation for the mean square generalization error of PINN. \n\nSecond, for Poisson equations on spheres or tori, PINN satisfies the convenient codiagonalization condition, which enables us to construct the examples, especially the worth-noting negative example in Section 5.2. \n\nExtending these results to other types of PDEs is both an interesting and important problem, and we plan to pursue this direction in future work. \n\nWe hope these clarifications help address your concerns. Thank you once again for taking the time to read our response and for your thoughtful feedback."}, {"review_id": "kyvf2SKzc3", "reviewer": "Reviewer_Aooh", "summary": "This paper establishes a necessary and sufficient condition for Physics-Informed Neural Network (PINN) consistency in solving second-order elliptic PDEs: convergence occurs if and only if the true solution $u^*$ lies in the closure $\\bar{\\mathcal{H}}$ of the NTK's RKHS under the PINN metric. The theory is proven via kernel gradient flow and NTK analysis, with non-asymptotic bounds derived under stronger assumptions. A pathological counterexample demonstrates PINN failure when the solution is not in $\\bar{\\mathcal{H}}$.", "questions": "1. In Corollary 3.11, is there a typo in $\\hat{u}_{\\theta(T)}$? I think it should be $\\hat{u}_T$.\n\n2. Is it possible to make Lemma 5.6 more quantitative?", "limitations": "The limitations are clearly discussed in Section 5.1 and Section 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**  \n- Establishes necessary and sufficient conditions for PINN convergence ($u^* \\in \\bar{\\mathcal{H}}_{NTK}$).  \n- Provides non-asymptotic error bounds under stronger assumptions.  \n- Pathological counterexample reveals fundamental PINN limitations.  \n\n**Weaknesses:**  \n- Verifying $u^* \\in \\bar{\\mathcal{H}}_{NTK}$ is intractable (requires NTK eigenfunctions).  \n- The experiments are only on a small scale.\n- Theory only applies to \"infinitely wide networks\" (NTK regime).", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Aooh sincerely for your feedback. We greatly appreciate the time and effort you have devoted to reviewing our paper. In response to your questions and concerns, we would like to provide you with the following answers and clarifications: \n\n1. Concerning the typo in Corollary 3.11: \n\nYes, it should indeed be $\\hat{u}_T$. We are very sorry for the typo, and will correct it in the revision. \n\n2. Concerning Lemma 5.6: \n\nYes, it can be reorganized into a more quantitative version. We thank the reviewer for this advice, and will include the quantitative version of Lemma 5.6 in the revision. \n\nFor your convenience, we provide a sketch of this quantitative version below. \n\nIt will have the following form: With probability $1-\\mathcal{O}(m^{\\frac{1}{2}})$, the two terms in (23) is in fact less than $Q$, where the quantity $Q$ depends on $m$, $N_u$, $N_f$ and $T$. The explicit computation of $Q$ is given by the right-hand side of (289) in Lemma H.3, and the terms $\\|A\\|_\\infty$, $\\|B\\|_\\infty$ and $\\varepsilon$ required in (289) can be found in (279), (280) and (281). Detailed computational results will be updated in the revised version of the paper. \n\n3. Concerning the intractability of $\\bar{\\mathcal{H}}$: \n\nWe thank the reviewer for raising this question. \n\nWe fully agree that the characterization of $\\bar{\\mathcal{H}}$ is crucial in the analysis of PINN. In this paper, we discuss several situations where the elements of $\\bar{\\mathcal{H}}$ can be completely characterized. \n\nThese situations, which might be of your interest, include: 1. The Sobolev RKHS $\\mathcal{H}=H^\\beta$, whose closure $\\bar{\\mathcal{H}}$ is precisely $H^2$ if $\\beta>2$; 2. The RKHS of the NTK on spheres or tori, whose closure $\\bar{\\mathcal{H}}$ is the Sobolev space $H^2$, as is shown in Theorem 5.9. \n\n4. Concerning the infinitely wide networks: \n\nWe thank the reviewer for raising up this question, and we fully understand your concern. Please let us provide you with some clarifications on this topic: \n\nUnderstanding the dynamics of neural network training is a fascinating and important topic, which we are currently exploring. The training process of networks can be viewed as a form of feature learning. In particular, training infinitely wide neural networks can be interpreted as performing feature learning with fixed, time-invariant features. We believe that investigating and gaining insights from the simplified setting of infinitely-wide networks may serve as a helpful first step toward understanding the more intricate behavior of feature learning for general networks. \n\nWith this motivation, our current work focuses on the infinite-width regime, in the hope that it may provide useful intuition and inspire further investigation into the training dynamics of PINNs in more general and practical settings.\n\n5. Concerning the small scale of experiments: \n\nWe thank the reviewer for the insightful advice and fully understand the concerns raised. In the revised version, we will include additional experimental results, particularly for PINNs applied to multi-dimensional PDE problems with more complex solution structures. \n\nWe hope these clarifications help address your concerns. Thank you once again for taking the time to read our response and for your thoughtful feedback."}, {"review_id": "asUNZyT6wK", "reviewer": "Reviewer_nmHw", "summary": "This paper consider the consistency of PINNs when applied to second order linear elliptic equations with Dirichlet boundary conditions. The authors establish the sufficient and necessary conditions for the consistency of physics informed kernel gradient flow and sufficiently wide PINN in general cases. The authors also construct a negtive example based on the theoretical results, highlights the risks of directly applying PINNs to specific problems without theoretical assessment of reliability.", "questions": "1. Can the results in this paper be extended to nonlinear elliptic equations, or time-dependent PDEs?\n\n2. Can the results be extended to discrete gradient flow? Notice that the convergence of continuous gradient flow and discrete gradient flow have been stated in [1] and related works.\n[1] Du S S, Zhai X, Poczos B, et al. Gradient Descent Provably Optimizes Over-parameterized Neural Networks[C]//International Conference on Learning Representations. 2018.\n\n3. What's the essential difference between this work and previous works? The authors give a short comparison in line 38-57, focusing on the training dynamics of networks, but how it affects the convergence result of PINN compared to previous works, especially [27] and [33] in your paper? A table comparison on the assumption, PDEs, convergence results ,etc, is suggested.\n\n4. What about the influence of initialization of neural networks? The paper suggest the parameters are initialized in a particular way in line 100-104, how about the commonly used Xavier initialization in codes, does it affect the results of this work?\n\n5. I have some doubts about the loss (2) $L$ and (3) $\\hat{L}$. While I think the main results should be about the emprical loss $\\hat{L}$, the continuous loss $L$ is frequently used in this paper. What comes to happen if we replace $L$ by $\\hat{L}$ in the main Theorem 3.7 and Lemma 4.1?", "limitations": "The authors addressed some limitations in the conclusion section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper study the dynamics of the training process and analyzing the convergence of PINNs with large network width.\nThe theoretical results is intresting and promising, show that the PINN solution depends on the true solution's smoothness and regularity.\nThis paper is clearly written, well organized, and a pleasant to read, contribute to the theoretical understanding of PINN community.\n\nWeakness:\n\n1. The results are constrained in seccond order linear elliptic equations with Dirichlet boundary conditions, limiting the theoretical guidence of PINN training for different kinds PDEs with different initial/boundary conditions.\n\n2. The concept of consistency and convergence are not well described for non-mathematicians. For AI conference, I suggest using the concept of convergence directly.\n\n3. The results are based on a continuous gradient flow of network training and neural tangent kernel. Practical gradient descent optimizers  are based on discrete gradient flow. However, this work didn't present results for discrete gradient flow.", "was_revised": true, "final_justification": "All my comments were well addressed. I will keep my positive score.", "rebuttal": "We thank Reviewer nmHw sincerely for your feedback. We greatly appreciate the time and effort you have devoted to reviewing our paper. In response to your questions and concerns, we would like to provide you with the following answers and clarifications: \n\n1. Concerning other types of PDEs: \n\nWe thank the reviewer for this question and would like to provide you with some clarifications on this topic: \n\n1.1. We are confident that the main results can be extended to other types of linear differential equations, including time-dependent PDEs such as parabolic equations. \n\nThe main results include Theorem 3.7, its application to network (Theorem 5.8) and the convergence of NTK (Theorem 5.4). The core derivations in their proofs remain valid for general linear equations. \n\n1.2. Extension to nonlinear equations is less immediate, because the loss $L$ and the empirical loss $\\hat{L}$ are no longer guaranteed to be convex with respect to the RKHS coefficients. This presents an interesting and important direction, which we intend to explore in future work. \n\n\n2. Concerning discrete gradient flow: \n\nWe thank the reviewer for raising this concern. Please let us take this oppotunity to offer some clarifications: \n\n2.1. The elegant results presented in [1] do not directly adapt to the PINN settings of our work, primarily for the following reasons: \n\nFirst, the training loss used in PINNs has a non-standard form that differs markedly from the classical mean-square loss, which gives rise to the unusual form of the training gradient flow of PINN (9). As we can see, the equation (9) is extremely different from the evolution equation of network in [1], because (9) requires computing derivatives of the kernel function and handling two distinct sample sets drawn from the boundary and the interior, respectively. Consequently, basic assumptions in [1] such as Assumption 3.1 (positive eigenvalue assumption for kernel matrix) is no longer readily verifiable for PINNs. \n\nSecond, the convergence result in [1] is established with respect to the empirical loss, whereas our work is concerned with the convergence of the continuous loss (population loss). \n\n2.2. In this work, we primarily focus on the continuous gradient flow, whose convergence relies on a delicate analysis of the ill-posed infinite-dimensional convex optimization problem. Whether an analogous statement holds for the discrete scheme is an interesting and very important problem. and we are actively investigating this direction with considerable interest. \n\n[1] Du S S, Zhai X, Poczos B, et al. Gradient Descent Provably Optimizes Over-parameterized Neural Networks[C]//International Conference on Learning Representations. 2018.\n\n\n3. Concerning comparison with previous works: \n\nWe thank the reviewer for this advice, and will add a table comparison of these references. More related references will also be supplied. \n\nIn particular, regarding the distinctions between our work and the references [33] and [27]—both of which the reviewer has kindly highlighted—please allow us to clarify: \n\n[27] investigates the convergence of the empirical minimizer of the PINN loss with H\\\"older regularization term for both second-order elliptic PDEs and second-order parabolic PDEs. The distinctions between [27] and our work are that: First, [27] only studies the empirical risk minimizer, and does not involve analysis of network training, while our work put the network training into consideration; second, the PINN loss in [27] is augmented with a H\\\"older regularization term, while our work uses the PINN loss without regularization. \n\n[33] investigates the spectral bias of PINN by examining the spectrum of the NTK of two-layer PINN for linear PDE problems with boundary condition (The term \"spectral bias\" refers to a well-known pathology that prevents deep fully-connected networks from learning high-frequency functions, as discussed in section 5 of [1]). The distinction between [33] and our work is that, [33] does not contain convergence result for PINNs, while our work provides the sufficient and necessary condition for the consistency of PINNs. \n\n\n4. Concerning initialization: \n\nWe thank the reviewer for raising up this question. This is indeed an interesting topic, and we would like to take this opportunity to offer some clarifications.\n\n4.1. The only purpose we use the special initialization (in line 100-104) is to ensure that the limit gradient flow estimator given by (10) lies in the RKHS of NTK at initial time $t=0$, so that the solution to the kernel gradient flow  (10) is well-defined. \n\n4.2. The choice of initialization is a delicate matter that needs careful attention in NTK theory. \n\nThe special initialization in line 100-104 is suggested by [1], and is popular in NTK related works. [1] also proved that the infinitely wide network will not lie in the RKHS of NTK at initial time, if all parameters are initialized as i.i.d. standard Gaussian random variables, in which case the NTK kernel gradient flow (10) will no longer be well-defined. \n\n4.3, Therefore, how the popular Xavier initialization has impacts on NTK theory is an interesting topic we are curious about and will perform study on. Thank you again for this valuable question. \n\n[1] Chen G, Li Y, Lin Q. On the impacts of the random initialization in the neural tangent kernel theory[J]. Advances in Neural Information Processing Systems, 2024, 37: 35909-35944.\n\n\n5. Concerning the loss functions: \n\nWe thank the reviewer for this question. We would like to offer some clarifications to address any potential misunderstandings. \n\n5.1. The empirical loss (or the training loss) $\\hat{L}$ is used solely to guide the training of network, whereas the population loss $L(\\hat{u})=d_{pinn}(\\hat{u},u^* )$ serves to evaluate the distance between the estimator $\\hat{u}$ and the true solution $u^*$ in the function space (The metric $d_{pinn}(\\cdot,\\cdot)$ is defined in (13) in the paper).  \n\n5.2. It is indeed possible to establish a convergence result for $\\hat{L}$, similar to Theorem 3.7 and Lemma 4.1, while it is a valuable but different story: \n\nOn the one hand, the convergence of the training loss $\\hat{L}$ to $0$ indicates that the network can be efficiently trained; \n\nOn the other hand, the convergence of $L$ to $0$ means that the estimator converges to the true solution in the function space. \n\nThank you for raising this point. We hope these clarifications help address your concerns on this topic. \n\n\n6. Concerning \"convergence\" and \"consistency\": \n\nWe thank the reviewer for this advice. We will take greater care in using the two terms in the revised version of the paper. Please allow us to offer some clarifications on our use of terminology: \n\n6.1. We have chosen to use the term “consistency” rather than \"convergence\" mainly due to the following considerations: \n\nFirst, our present work focuses on the consistency of PINNs, while we plan to investigate their convergence rate in future research. Thus, we adopt the term \"consistency\" from learning theory in order to distinguish this work from the existing and forthcoming results that address convergence rates. \n\nSecond, the term \"consistency\" emphasize more on the convergence of the population loss $L$, and we hope to distinguish the convergence results of the population loss in this work from the convergence results of the training loss $\\hat{L}$. \n\n6.2. We will replace many instances of the term \"consistency\" with \"convergence\" throughout this paper, including the title, to avoid potential ambiguity and confusion. We will also provide clearer and more precise explanations for the term \"consistency\" in the revised version of paper. Thank you again for this valuable advice. \n\n\nWe hope these clarifications help address your concerns. Thank you once again for taking the time to read our response and for your thoughtful feedback."}]

Author rebuttal:

Meta-review: **Summary**  
This paper investigates the consistency of Physics-Informed Neural Networks (PINNs) when applied to solve second-order elliptic partial differential equations (PDEs) with Dirichlet boundary conditions. The core contribution is the establishment of a necessary and sufficient condition for the consistency of the PINN method, framed within the infinite-width Neural Tangent Kernel (NTK) theory. This condition states that a PINN is consistent if and only if the true PDE solution lies in the closure of the NTK's Reproducing Kernel Hilbert Space (RKHS) under the PINN metric. The paper further derives non-asymptotic loss bounds under stronger assumptions and constructs an insightful pathological counterexample that demonstrates the failure of PINNs when this condition is not met。

**Strengths of the Paper**  
1. The paper addresses a foundational question regarding the reliability of PINNs by being the first to establish a necessary and sufficient condition for their consistency. This lays a solid theoretical groundwork for understanding when PINNs can be expected to work.
2. The study employs a rigorous mathematical framework, leveraging kernel gradient flow and NTK theory to analyze the training dynamics of PINNs.
3. The authors construct a concrete example where the PINN method is inconsistent, which clearly illustrates the practical implications of their theoretical findings and highlights a key failure mode of PINNs.

**Weaknesses of the Paper**  
1. The paper's most significant initial weakness was its focus on shallow networks, which limits practical relevance. While the authors have provided a credible proof sketch for extending the results to DNNs, the full proof has not yet been peer-reviewed
2. The analysis relies on the infinite-width NTK regime and a continuous gradient flow, which are idealizations of practical finite-width networks trained with discrete optimizers. Furthermore, verifying the core assumption about the true solution's membership in the RKHS closure can be intractable in practice.
3. The experiments presented were noted to be on a small scale, and the authors have been encouraged to include more complex, multi-dimensional problems.

**Summary of the Review Process**  
The paper was thoroughly evaluated by four reviewers, with final ratings consisting of one "Borderline reject" (3) , one "Borderline accept" (4) , and two "Accept" (5). All reviewers acknowledged the paper's theoretical rigor and novelty , particularly appreciating the fundamental nature of the necessary and sufficient condition and the insightful counterexample that clarifies the limitations of PINNs.  

The primary points of concern revolved around the scope and applicability of the theoretical analysis. The most critical weakness, raised by Reviewer VWqv, was that the initial analysis was restricted to two-layer (single-hidden-layer) networks, limiting its practical relevance as deep networks are dominant in practice. Other concerns included the analysis being confined to a specific class of PDEs , the reliance on strong assumptions that may be difficult to verify (e.g., the solution lying in a specific Hilbert space) , and the idealization of using an infinite-width network in a continuous gradient flow setting.  

The authors provided a very detailed and technical rebuttal. To address the critical issue of multi-layer networks, they stated that a proof for deep neural networks (DNNs) has been completed, and they provided a comprehensive proof sketch to support this claim. They also offered sound justifications for other points, such as the rationale for focusing on second-order elliptic PDEs and their plans to expand the experiments in the revised version.  

The authors' rebuttal was successful in convincing the majority of the reviewers. Reviewers yJxF and nmHw maintained their high scores after the rebuttal, and Reviewer Aooh also expressed satisfaction. The sole exception was Reviewer VWqv, who, despite the detailed response, remained unconvinced about the extension to DNNs and maintained their "Borderline reject" score.


**Recommendation: Accept**  
Despite some limitations, the theoretical contributions of this paper are significant and foundational. It provides deep insights into a core question in the important and growing field of PINNs. The authors addressed the reviewers' main concerns convincingly in their rebuttal, particularly the critical point of extending their theory to deep networks, for which they provided a detailed technical sketch. While one reviewer remains skeptical, the overall consensus is positive.
Decision: Accept (poster)