How does PDE order affect the convergence of PINNs?
Authors: Chang hoon Song
Yesom Park
Myungjoo Kang
Venue: NeurIPS 2024 poster (2024)
Keywords: Physics-Informed Neural Networks
Variable Splitting
Higher-order PDEs
Gradient Flow
Convergence
Primary area: machine_learning_for_physical_sciences

Abstract: This paper analyzes the inverse relationship between the order of partial differential equations (PDEs) and the convergence of gradient descent in physics-informed neural networks (PINNs) with the power of ReLU activation. The integration of the PDE into a loss function endows PINNs with a distinctive feature to require computing derivatives of model up to the PDE order. Although it has been empirically observed that PINNs encounter difficulties in convergence when dealing with high-order or high-dimensional PDEs, a comprehensive theoretical understanding of this issue remains elusive. This paper offers theoretical support for this pathological behavior by demonstrating that the gradient flow converges in a lower probability when the PDE order is higher. In addition, we show that PINNs struggle to address high-dimensional problems because the influence of dimensionality on convergence is exacerbated with increasing PDE order. To address the pathology, we use the insights garnered to consider variable splitting that decomposes the high-order PDE into a system of lower-order PDEs. We prove that by reducing the differential order, the gradient flow of variable splitting is more likely to converge to the global optimum. Furthermore, we present numerical experiments in support of our theoretical claims.

How does PDE order affect the convergence of PINNs?
Changhoon Song
Yesom Park
Myungjoo Kang²
Research Institute of Mathematics, Seoul National University Department of Mathematics, University of California, Los Angeles
Department of Mathematical Sciences, Seoul National University changhoon. wwwgs3ogmmil...oooooooo com, yeisom@math. ucla. edu, mkangøsnu. ac kr
Abstract
This paper analyzes the inverse relationship between the order of partial differential equations (PDEs) and the convergence gradient descent in physics-informed neural networks (PINNs) with the power of ReLU activation. The integration of the PDE into a loss function endows PINNs with a distinctive feature to require computing derivatives of model up the PDE order. Although it has been em- pirically observed that PINNs encounter difficulties in convergence when dealing with high-order or high-diig nigh-dimensional PDEs, comprehensive theoretical under- standing of this issue remains elusive. This paper offers theoretical support for this pathological behavior by demonstrating that the gradient flow converges in lower probability when the PDE order is higher. In addition, we show that PINNs struggle address nigh-dimensional problems because the influence of dimensionality on convergence is exacerbated with increasing PDE order. To address the pathology, we use the insights garnered to consider variable splitting that decomposes the high-order PDE into system of lower-order PDEs. We prove that by reducing the differential order, the gradient flow of variable splitting more likely to converge to the global optimum. Furthermore, present numerical experiments in support our theoretical claims.
Introduction
Understanding of partial differential equations (PDEs) fundamental describing diverse phenom- ena in science and engincering, including fluid dynamics [60] weather prediction [5I], disease progression B [6)). and quantum mechanics [20 g. This underscores the imperative necessity for the effective acquisition of their solutions. Given that analytically solving PDEs often infeasible even impossible for numerous practical scenarios due to their complexity, numerical methodologies play pivotal role in approximating solutions to PDEs, enabling researchers and engineers to address real-world problems effectively.
The advent of deep learning has led to surge in attempts leverage it to solve PDEs 59 42
30
Among these, physics-informed neural networks (PINNs) [1 38 58) stand out as prominent methodology. Coupled with the automatic differentiation technique 77.. they integrate the residuals of PDEs and boundary conditions into the loss function, thereby enforcing the approximation of solutions using artificial neural networks. This distinetive incorporation of PDEs into the loss function introduces partial differential operators in calculating the loss, distinguishing PINNs from conventional deep learning models. Renowned for their accessibility and versatility in being capable of easily handling arbitrary PDBs and being mesh-free, PINNs have garnered significant attention
and demonstrated promising outcomes across various fields 1 28 65
Equual contribution authors. Correspondence to: <mkang ackr..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite their potential, PINNs frequently encounter difficulties in accurately approximating solutions, particularly when the governing PDE contains high-order derivatives 48| 33). They also exhibit sensitivity to increasing dimensions 33 These challenges impede the practicality of PINNs due to the pervasiveness of high-order or high-dimensional PDEs numerous physical and engineering
descriptions, such as control problems [22 57]. finance [ 53), phase separation [2 27], and
mechanical engineering [4 32]. Several studies have indicated that neural network architectures possess sufficient expressive power to approximate solutions 41). However, it has been purported that the inferior performance may be attributed to the difficulty in optimizing PINNs, which arises
from including the PDE in the loss function [37 61 62). Despite the widespread use of PINNs, rigorous mathematical understanding of these pathological behaviors of PINNs has been lacking.
In this paper, we endeavor to provide mathematical understanding of the pathological behaviors of PINNs by analyzing the convergence of their gradient flow (GF), which reveals a profound sensitivity of the GF convergence with respect to the PDE order and the power of the activation. Building upon the work of Gao al. 255 we extend the analysis of the GF of PINNs, composed of two-layer multilayer perceptrons (MLPs), to general kth-order PDEs and the p-th power of Rectified Linear Unit (ReLU) activation function with general p. We achieve tighter bounds than those obtained by Gao et al., shedding light on the underlying causes of the pathological behaviors of PINNs. Our theoretical findings demonstrate that the width size of the network necessary for the convergence of the GF increases exponentially with the power p of ReLUPactivation. Furthermore, our results indicate that the optimal power p is determined by the order i of the governing PDE, specifically to be k + 1. We also find that the PDE order impedes the convergence of GF, where this negative impact of the PDE order stems from incorporating the PDE into the PINN loss function, which necessitates network differentiation up to the order of the PDE. Moreover, our theoretical investigation unveils that the GF convergence of PINNs also deteriorates with increasing dimensions, and the differential operators included in the PINN loss further exacerbate the sensitivity of PINNs to dimensionality. This elucidates why PINNs are relatively sensitive to dimensionality compared to conventional deep learning models that do not involve differentiation in the loss function.
To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy [54] (55 66, which represents derivatives of the solution additional auxiliary variables. The key point of variable splitting is that learning a high-order PDE boils down to learning a system of lower-order PDEs. Reducing the order of derivatives included in the loss function, the strategy alleviates the difficulties associated with the PDE order. It further enables to utilize more general ReLuacitiationn with lower power p than PINNs The lower differential orders that the network computes, the more likely it is that the GF will converge, so the most suitable one among the various constructions of the variable splitting method is the finest splitting, which separates all the derivatives into auxiliary variables and reformulates the PDE into a system of first-order PDEs. This strategy results in loss function comprising only first-order derivatives, and the efficacy of this finest variable splitting would be magnified as the order of the governing PDE or dimension increases. Therefore, the finest splitting approach would exhibit a pronounced discrepancy from the vanilla PINNs for high-order PDEs. Morcover, a reduction in the differential orders enhances the resilience of the model with respect to dimensionality. Finally, we present numerical experiments to verify our theoretical findings and validate the effectiveness of the variable splitting.
1.1 Related Work
Characterization of Gradient Descent for PINNs As significant issues have been identified within physics-informed machine learning, numerous mathematical studies have been conducted to elucidate the behavior of PINNs. While studies have been mainly dedicated to examining the generalization capacity of PINNs 19 49 23), there has also been work on understanding the difficulty of optimization, which is believed to be the primary source of failure for PINNs. Wang et al. [61] found that PINNs exhibit stiff gradient flow dynamics, resulting in imbalanced gradients during training. Ryck et al. [18] characterized the rate of convergence in terms of the conditioning of an operator and suggested that the difficulty of training PINNs is closely related to the conditioning of the differential operators in the governing PDEs. Another work [62] utilized the neural tangent kernel (NTK) theory to indicate that spectral biases and discrepancies between convergence rates of various loss components can lead to training instabilities. Global convergence properties of PINNs for second-order linear PDEs have also been studied within the NTK regime [4 and using the
Rademacher complexity 47 Most closely related to this paper, Gao et al. 25 demonstrated the

---

convergence of the gradient descent for two-layer PINNs. However, their discussion is limited to second-order linear PDEs. We extend the analysis general kth-order linear PDEs and p-th power of activation functions and provide tighter bounds than Gao et al.. These advances allow us to observe further the effect of the order and dimensionality of the PDE on the convergence.
Variable Splitting The method of separation of variables, which simplifies differential equations by reformulating them into more manageable system, is a classical method for solving differential equations [9]. In particular, it has been widely employed when dealing with high-order PDEs as augmenting high-order derivatives as additional variables allows the governing equation to be decoupled into set of lower-order PDEs that are comparatively easy to solve [16 26 63). Recent
endeavors have explored the integration of separable variables within the PINN approach. In this
paper, we refer to this approach as variable splitting according to [54] [5 56). Augmented variables
have been introduced to represent vorticity in the Stokes equation (6 the gradient of the solutions
for solving the p-Poisson equation [54], and the eikonal equation SS Additionally, second-order derivatives have been separately parameterized to solve bi-harmonic equations effectively [45]. The rationale for introducing auxiliary variables in previous works is to enhance the efficiency and accuracy of PINNs, but they lack a comprehensive theoretical clucidation of its effect. A recent study [56] has theoretically analyzed variable splitting, demonstrating that while PINNs do not guarantee convergence to the PDE solution even when the loss converges to zero, variable splitting does ensure convergence to the solution for second-order linear PDEs. In this study, analyze the impact of variable splitting for PINNs with ReLU activattio in terms of the convergence of the GF.
1.2 Main Contributions
The contribution of the paper is summarized follows.
We analyze that the GF of PINNs with ReL.UPactivation converges to the global minimum for general kth-order linear PDEs. This extends the findings of Gao et al. (25) to encompass broader range of PDEs and activations and provides
tighter bound.
We demonstrate the inverse relation between PDE order and the GF convergence, unveiling the adverse effect of the differentials included in the PINN loss the GF convergence.
We provide a theoretical understanding of the reasons why PINNs encounter difficulties in addressing high-dimensional problems.
We prove that the order reduction of variable splitting, which reformulates the PDEs into system of lower-order PDEs, results the convergence enhancement of GF.
2 Mathematical Setup
Arbitrary Order Linear PDEs We consider a general form of kth-order linear partial differential equations (PDEs) defined on a bounded domain Ω C Rdd (in which the temporal dimension could be a subcomponent)
f (z),
x E Ω a E an,
8 where N [u] = Jalsk Ga Dare is a kth-order linear differential operator with coefficient functions
aa : Ω + IR for each multi-index a E No. B [u] = Σ Eaass āa - u represents the boundary condition operator with coefficient functions a, : an - R, which could reflect Dirichlet, Neumann, and Robin conditions
{ ʃ Ω IR is given source function, and g an IR is a given boundary function, and u Ω + IR. is the unknown solution of interest.
"The boundary condition for high-order PDEs is typically given by multiple conditions of higher order than one. Our approach also encompasses such general boundary conditions by incorporating the residuals of each boundary condition into the loss. The sole distinction is the utilization of induction not only on the derivative matching losses but also on boundary losses to prove Proposition 2. For the sake of brevity, we assume that the boundary condition is the most prevalent (weighted combination of) Dirichlet and Neumann conditions.

---

Physiss-IffrmeedNeurall Networks proximate the solution u of the PDE by neural networks. Following the prior work [25,, approximate the solution by two-layer multi-layer perceptron Φ Rd - I of width m, defined as
neural networks (PINNs) [88] aim to ap-
(x; w. v)
(w, y),
(2)
where w, E RR+2 v, IR, xx 2] E RR+1, and (-) is the activation function. For brief notations, assume that Q is bounded so that ////|2 < 1 for æ: E Q. We consider the case where is the ReLU activation function for an integer p. which also known as Rectified Power(RePU) activation [ [ 15 As it will be clear in the context, the power p necessitates surpassing the order & of the PDE T to ensure that the loss function and gradient descent flow are well-defined. Therefore, our analysis is focused on scenarios where p 2 ++1 1. PINNs learn the parameters of by minimizing composite loss function, comprising the residual of the PDE and the boundary condition (L) which enforces the network's compliance with the governing physics. For given the training data {æ,, J (xx)) C Ω R and { y (i,)) Th
[ :........
Rmld+2) v [10 m R y
j=1 02 x IR of respective sizes n, N and E N. PINN loss function is given by
CPINN (w, = 2 (Iss (w., uw + 1h (w, w)1l)
(3)
where 8 (w,v) v) = [s1 (w, v)
... Sna (w, w) and h (w, v) = [h1 (w. ... hns (w, D)] with (w, v)
(W (6 (;w, w)] (x.) (xx)),
(4)
h (w, v) (B 1 (: w. w)] (ā,) - (iz)),
(5)
and v > 0 is a regularization parameter that relatively balances the two components of the loss.
Gradient Flow As the limiting dynamics the gradient descent (GD) with infinitesimal step-sizes
40
gradient flow (GF) is continuous time dynamics that starts at u (0) and v (0) and evolves as
dua (t)
DCPiNy(w.s) @CPiNy(M..) dv,
> Tha i=1 8; (w, v)
0s. (s) Ow. (s,(10
Σ 116 j=1 hy (un, v)
0h, (w.w)
a (w.u) (6)
Ow
(u..(t) dt
E.e 8; (wn v)
ΣØ=1 hy (un, v)
Sv.
Initial weights are supposed follow the normal and uniform distributions, ur (0) N (0, Jm) and ~ U ([-1, 1}), respectively [3] GF can be regarded as a continuous-time analog of GD and is frequently employed to comprehend the behavior of GD optimization algorithm in the limit. By the chain rule in conjunction with (6). the following characterizes how the loss function evolves during training by gradient descent:
s (w (t). di (w (1), [8] = - (w (t), (()) + G. (w (1) (1), (t))
s (w (t). (t))
h (1),
(7)
where G and G are Gram matrices for the dynamics, defined by
Gu (w, v) = Dw. Dw
dsg (w. v)
Bena 8u
(w. v)
(w. v)
ahun (w, v)
(8)
G. (w, D.. D. =
Øs1 dh
(w, v)
8snu (w, v)
0h1 (w, v)
Bhng th
(w, v)
We are interested in analyzing the effect of the PDE order on the convergence of the PINN loss, which evolved in accordance with the dynamics 77). to the global minimum zero.
Indeed, our analysis covers more general initialize distributions than 25 In our analysis. it is enough that probability density function of w (0) is in Schwartz space, and that of e (0) is bounded and has expectation.

---

3 Impact of PDE Order on Convergence of PINNS
Despite the demonstrated promise and versatility of PINNs in addressing wide range of problems [29 44 36.. they often encounter difficulties in constructing an accurate approximation to the desired solution of PDES, particularly with high-order PDEs. Moreover, in contrast to the confirmed efficacy of neural networks in modeling high-dimensional data such as images and text, the exploration of PINNs for high sigh-dimensional PDEs has been apparently limited. While neural network architectures possess sufficient expressive power to approximate solutions [I]] inferior performance has been attributed to the difficulty in optimization in practice [6T] 62 Additionally. it has been postulated that the optimization difficulty may stem from the partial differential operators included in the loss Nevertheless, despite the significant challenge posed by these pathological phenomena, there remains a paucity of theoretical understanding of them.
function [37 48 33
In this section, we theoretically elucidate these pathological phenomena by studying the convergence condition of GF 7 of PINN loss (3). Specifically, we provide a width condition for 77) to converge global optimum in terms of order k, dimension d, and the power p of ReLU activation. Analyzing how those factors are related to the convergence condition, we explain why optimizing PINNs is harder when the order or degree higher.
Following [4] and (25. we first prove the positive definiteness of the limiting Gram matrix of PINNs for general kthhorder linear PDE and p without any further strict assumption other than p > k.
Proposition 3.1 (Special Case). The limiting Gram matrix G
-Eur. [G (w, v)]] is strictly
posilive definite and independent of m.
This is a special case of the general state in Proposition C.3 with L = 0, and the proof for the general case is provided in Appendix C We denote the smallest eigenvalue of G by Ào > 0. The following presents our main theorem in this section, the requisite width size m for the GF of PINN loss to converge the global minimum with high probability. The result demonstrates that the required width grows exponentially as the PDE order k and the dimension of the domain d increase.
Theorem 3.2 (Special Case). There exists a constant C, independent of d, k, and p. such that for any 6 < 1, if
> C d kk+kk> 14 p 2k4-4200
md
Ap
logg
(10)
d
8
then with probability of at least 1 - 8 over the initialization, we have
LPNNN (w (t), v (t)) S exp (-Aot) CPINN (w (0) v (0)), Vt 2 0.
(11)
It is a special case of Theorem 4.3 with L = O, the proof of which can be found in Appendix C.I It extends, inspired by (25). the convergence of the GF of PINNs of second-order linear PDEs to kth-order linear PDEs and the general p-th power of ReLUU. It states that even in these general settings, the GF of PINNs converges to the global minimum with a high probability when the width of the
network is sufficiently large. Moreover, we obtain a polylogarithnin bound (log !) áp which is much tighter than polynomial bound 8 in (25) for 3. These improvements permit the derivation of the following valuable explanations for the deficiencies observed when optimizing PINNs.
sheds light on the suitable choice of activation function for PINNs. In the training process, the activation function plays an important role. However, there are no clues as to which activation function is favorable to the given optimization process. Especially in the case of PINNs, it depends heavily on the PDE at hand. Despite its pervasive use in deep learning due to its numerous advantages and performance benefits, the ReLU activation function not admissible in the PINN framework, which necessitates the activation function to provide high-order derivatives for optimizing PDE-based constraints. Instead, PINNs harness the p-th power of ReLU as the activation function. It is apparent that p must satisfy p 2 + 1 for the PINN loss and gradient descent to be computed. Theorem 3.2 indicates that the smaller p is, the more likely the gradient descent will converge; that is, it is most optima to adjust p + 1 regarding the training process.
Optimal Power of ReLU Function in Training Theorem sheds
'In terms of approximating the solution of the PDE, larger makes the network smoother and has better expressive power [I]] However, it means that the set of networks covers broader function spaces p increases,
5

---

Understanding Difficulty in High-order PDEs A significant observation of Theorem 3.2 is that provides a theoretical understanding of why PINNs struggle with high-order PDEs. From (TO). we can see that the bound increases exponentially with the order of the PDE. Moreover, for the GF of PINNS to converge with high probability, that is, 8 < 1, a small increment of the power p would contribute to non-negligible degradation in the convergence, which could ultimately prevent the network from reaching a minimizer of the loss. Hence, given that & determines the admissible p by p 2 k + I, the order k of PDE primarily influences the convergence of PINNs and increasing the exponential term in (TO).
Understanding Difficulty in ligh-diimesional Problems The above theorem, which shows that the lower bound of m depends on the exponential of d, explains why PINNS cannot completely combat the curse of dimensionality, As PINNS are regarded as a versatile method capable of being mesh-free, they have been expected to be free from the curse of dimensionality [58]. However, the GF of PINNs becomes harder to converge d increases, requiring the network to be wider. Furthermore, it can be observed that the magnitude of change in d is amplified with respect to the exponent of k. This explains why PINNs are relatively sensitive to increasing dimensionality in comparison to other deep learning models whose loss functions do not contain derivatives. In other words, the presence of derivatives the loss makes PINNS sensitive changes in dimensionality. and the larger k is, the more difficult PINNS are for high-dimensionality.
Combining all crucial observations from our main theorem, we believe that the impact of the PDE order is one of the primary underlying reasons why PINNs often fail to minimize their loss. In light of this theoretical evidence, the next section describes variable splitting strategy that addresses these pathologies by properly reducing the differential order in the PINN loss function.
4 Order Reduction through Variable Splitting
The previous section indicates that the PDE order l significantly affects the width requirement for the GF to converge. Concurrently, for kth-order PDEs, is necessary to increase the ReLU activation to at least the k: + power in order to ensure a well-defined GF for the PINN loss. Consequently, lowering i could potentially lead to better convergence of the GF. In this section, we introduce variable splitting strategy to decrease the differential order by reformulating the given PDE into system of lower-order PDEs. We then extend Theorem 3.2 to more general form in Theorem
4.1 Variable Splitting
The concept of variable splitting 54
is to rewrite higher-order PDE into lower-order system, after which the PINN approach is applied to the system. A crucial aspect of the success of such methods is the reduction of the derivative order present in the training loss function.
55 56
Augment Variables For L 2 0 and increasing integers 0 = Eo < & <... < EL+1 k. variable 88 splitting augment the derivatives of the solution Buit U, .... a u as additional auxiliary variables
d1. ......, Φr. respectively. For notational simplicity, we abbreviate the integer set (1....,. m) for g+r u for 0z 2F
positive integer m by [m]. For E E [L], each term in
corresponds to
multi-index a = (an,..., Caa) E Nò with the size la) = >d
=- (4 = n. Therefore, Φe is a vector-
valued function of size [Tel for the index set defined in We denote the component of de that
O&e corresponds to Danc by (de)a:
8 Reformulate PDE into Lower-order System By replacing each of the differential terms 3z with the corresponding auxiliary variables dr, the differential operator / in can be rewritten as:
go
N [
Σ loi
aa ax V ΣΣ Σ
8x (Φt)
(12)
E= JO|SEE 18ISAEE1
not that network can be easily trained. Indeed, our result leads us to the opposite conclusion that large could be detrimental to convergence from the optimization perspective. In this paper. we refer to 'optimal" as the sense of being likely trained under the mildest condition rather than approximating the solution with the smallest error.

---

for some coefficient functions a 8 : - IR and A &e - Şe-1. Since Oe represents a function that differentiates the PDE solution AEg-times more than 1 the components of two consecutive variables Φ and Φ are governed by D
(de-1), 1). (a) (dt) (æ), a E Lec-1. 3 E Late
(13)
dx
From these, the PDE ① can be identically reformulated by the system of lower-order PDEs:
N ou, ΦL) (x) f (a)),
x EEΩ,
Pm ( (z) = (0t) (x), B [00] (x) 9.
x E Ω2 f E [L], a E la-- B LaE
(14)
a an
It is of paramount importance note that the maximum differential order of this system of PDEs is the highest difference of derivative order between consecutive auxiliary variables [cl = max {AEz: L E [L + 1]}, which is less than k. This aspect gives rise to notable ramifications in analysis of VS-PIN which will be discussed in the next subsection.
Variable Splitting for PINNs this paper, we consider the parameterization all variables de with two-layer MLPs with ReLU activation function, in manner analogous to that described in Section 2 for PINNs. The weights in the first and second layers of Φr are denoted by we and uy, respectively. We use of all weights. Similar to PINNs, Variable Splitting for PINNS (VS-PINNs) employ the linear sum of penalized residuals of each term of the induced system of PDEs (14) as the training loss:
[w] ...,,,]] and v [v] .... to refer to the respective collections
1 Tho
CVS PINN (w,y)
N ................................ ((o)....,))]((a (1:))
O Oas (oc- 1) (x.) (0t)
VE
Σ
(15)
No
Sa
1815
(B [dn] (āt,) 9 (0)
71b
where v, My....,P regularization parameters. As the GF of LPINN is characterized by Gram matrices G and G, induced from the gradients of the residuals of each term ① the GF of CysN is characterized by Gram matrices G and Go, which is induced from the gradients of the residuals of each term in 114 Appendix A gives more details for G. and G..
Remark 4.1. In order for high-order PDEs with k > 2 to be well-posed, it is necessary to have more boundary conditions than those defined by the boundary operator B in (T. Although our analysis concentrated B that reflect only up to first-order derivatives for the sake of simplicity, our theory can also be applied to more general boundary conditions. Furthermore, the high-order boundary conditions B can also be reformulated using the auxiliary variables used for N. In that case, relations (33) should hold on the boundary x E US2 As the reduced system (14) with reformulated boundary condition is equivalent to (I). instability issues were not observed in our numerical experiments even in the absence of artificial boundary conditions on the auxiliary variables unlike grid- based conventional numerical schemes.
4.2 Analysis
A key advantage of VS-PINNs is that the derivative order of the induced system of PDEs 14 is [g), which is lower than that of the original PDE (I) We prove its effectiveness in this section. As analogous to PINNs, providing its proof in Appendix C
begin by proving the positive definiteness of the limiting Gram matrix,
Propusition 4.2 (General Case). The limiting Gram matrix G Ew.v [G (w, e) is strictly positive defmitt and independent of m.
We denote the smallest eigenvalue of G by Ao > O. We now present our main theorem, which demonstrates the profound impact of order reduction variable splitting. The proof of the following theorem can be found in C
7

---

Theorem 4.3 (General Case). There exists a constant C, independent of d, k. (E), and such that for any d < 1, if
d d+ k) ( (+=1)))))) p 2 m C
mdd) 4p
log 8
(16)
d
then with probability of at least 5 over the initialization, we have CY PINN (w (t), (t)) I exp (-Aot) CPİNN (w (0), v (0)), Vt 2 0.
(17)
The right-hand--ie of (16) grows exponentially with respect to d. k. and p. thereby indicating the substantial influence of these factors on the convergence of VS-PINNs, including PINNs specific case (L = 0). This analysis reveals several significant advantages of VS-PINNS:
Improved Convergence: VS-PINNs are more likely to converge to the global optimum than PINNs
due to the reduction in the derivative order [|] < k. This also relaxes the condition on y from p 2 k+1 to p 2 [6|+ + 1. As previously discussed in Section 3 the optimal value of p is [S] + 1. Given that 6 (1E) + 1) is exponent of log (1) the most dominant term, reducing the order from & to [ leads to an immense improvement. There is another noteworthy observation we can see here. Given a kth-order PDE, there numerous possible partitions & that could be employed to decompose it to a system of lower-order PDEs. Consequently, there are a many of potential VS-PINNs that could be constructed. The aforementioned result indicates which of these is the most effective. As the convergence improves dramatically with a reduction in the derivative order, the optimal approach for splitting variables among various ways is to separate the given PDE into system of first-order PDEs by parameterizing all derivatives of the solution as auxiliary variables. In other words, the finest splitting with Eo- = 0, E ..... Čk-11 1 k would be the most effective in terms of the convergence of GF, as the differential order |s] is reduced the most to 1. Taken all together, the most optimal VS-PINNs that reduce the PDE order &o to 1 will markedly enhance the convergence of GF.
The reduction of orders in VS-PINNS enhances the resilience of the model to -ige-dimnnsionaltyy. From Theorem 3.2 we observed the effect of d being exponentially enlarged for the PDE order is due to the leth-ordrr partial differential operators in the loss function. It can be alleviated by VS-PINNs reducing the order, thereby easing the amplified scale to exponential of [g]. This indicates that VS-PINNS are more effective in combating the curse of dimensionality. Since the curse of dimensionality is a serious issue that is prevalent in various fields, including Hamilton-Jacobi-Bellin
Reduced Dimensional Impact:
equation in control problems, Schrodinger equation in quantum physics, and Black-Scholes equation in finance, it is evident that enhancements to the robustness of VS-PINNs with respect to their dimensionality would facilitate considrralle advancements in various fields.
Memory Efficiency: VS-PINNs are memory-efficient despite the presence of multiple auxiliary networks. As the order of the derivative increases, the complexity in automatic differentiation in modern deep-learning frameworks like PyTorch increases and it becomes computationally expensive 77. Adopting the order-reduced representation in the proposed variable splitting can overcome the difficulty in calculating the high-order derivative via automatic differentiation. The loss function for the finest VS-PINNS involves only first-order derivatives, which reduces the memory usage and computational requirements. Despite the increase in the number of networks, VS-PIÑNS exhibit greater efficiency because memory usage and computation scale linearly with the number of networks in contrast to the exponential scaling with the order of derivatives. Table 3 in appendix demonstrates the memory reduction of VS-PINNS.
Remark 4.4. The current approach to parameterizing the &--th order differential operator on all axes & as an auxiliary variable may be suboptimal in certain cases. In a given PDE, if the order of the derivative varies significantly along the axes, that is, aa 0 for only a few in [ it may be more efficient to approximate the partial derivatives using auxiliary variables separately for each axis. To illustrate, for the PDE uu Ugarz it is more suitable to parameterize variables on as d1 Fi (Ut, Mg). Φ2 R Ugz, and dy R3 Mzza, rather than approximating all tensors (t.x)) V2 (taa) and (1. The theoretical framework presented in this paper is capable of addressing this scenario by constructing each dh to replace Swn the intricate nature of the states and the lack of meaningful impact on the PDE order.
géc for only part of a with |a| = Ee. However, we exclude it due to
8

---

10
10
13a
(a) Convergence behavior of PINNs on bi-harmonic equation
2.-6
-00
10
10
3
m=-12
--1
10:00
n=11
0-2
1r12
10:10"
Epocils
Epocts
Epoctss
Epocts
(b) Convergence behavior of PINNs Poisson equation
Figure 1: Training losses of PINNs solving (a) bi-harmonic equation and (b) Poisson equation.
Remark 4.5. Although the shaprness of the bound in Theorem 4.3 is open, it is important to note that the leading term of the bound is based on conditions necessary for the Gram matrix to be positive definite, which is a crucial property of the Gram matrix for ensuring the convergence of the GF to global optimizer. Since the Gram matrix defined by the PDE loss and the network structure, we believe it can still provide valuable insight into how order and power affect convergence.
5
Experiments
This section presents experimental results that validate the theory. Throughout numerical experiments, two-layer MLPs with ReLUPactivation function were utilized in order align with thcoretical framework. Throughout all experiments, the training collocation points consists of uniform grid and regularization parameters set to 1/.....; VL 1 and 10. We implement all numerical experiments on a single NVIDIA RTX 3090 GPÚ. Experimental details are provided in Appendix D
Convergence behavior of PINNS investigate the influence the activation order p and the PDE order k on the width required for convergence, examined both the second-order Poisson equation and the fourth-order bi-harmonic equation, both of which yield the same solution. We trained networks with varying widths ranging from 10 to 10%, for each combination of and k using GD optimization with a learning rate of 10-8 Figure [ illlustrates the training losses at the initial stage a logarithmic scale, supporting theoretical findings that larger width is needed for higher values of p to ensure convergence. Moreover, we can observe that narrower networks tend to converge more readily when solving lower-order PDEs (Poisson) compared to higher-order PDEs (bi-harmonic). This observation aligns with Theorem 4.3 that higher-order PDEs necessitate larger network widths for guaranteed convergence.
Validation on the effect p To verify the influence of the power of the ReLU activation function, we test PINNs with varying values between 3 and 10. Since the training process became highly unstable p increases, we consider second-order heat equation (509) [14] gain more precise
investigation of the effect of p. The results are summarized in Figure 2 (a). We can see that the convergence of loss is enhanced as p decreases, which supports our theoretical finding.
To validate the order reduction effect of VS-PINNs. we conducted an experiment comparing PINNs with VS-PINNs on the second-order heat equation. Each model was run five times with different random seeds, and Figure 2 depicts the training loss for both PINNs and VS- VS-PINNs along with their variance. The results show that the training loss for VS-PINNs converges more effectively than that of PINNs. This indicates that VS-PINNs. which optimize a loss function incorporating lower-order derivatives using networks with smaller p. facilitate convergence of GD, consistent with the theoretical findings in Section 4 Furthermore, performed a similar experiment on the convectionddiffsiinn equation (51T) in the Appendix and obtained results that were consistent with those observed for the heat equation.
Comparison between PINNs and VS-PINNs
Effect of splitting level For higher-order PDEs, there are several ways transform a given PDE into a lower-order system through variable splitting. To investigate this effect, we conducted experiments on the fourth-order clastic beam equation 510 [52] with two cases: (i) do Rs u, 01 Rs ue uxz

---

Train lossus for I-D heat equation
Heat eqution
Boan equztion
10.0
Fibes.
VS RENN
VS- 15)
1.0
VS-TINN
Toain loss
Triin loss
0.01
0.001
D-7
0.01
0.0001
p-9
1e:05
p-10
0.001
2000
3500
4000
500
1080
2000
3000
3550
4000
(x1)
Epoch (x100)
Spoch 6x100)
(a) Effect of p
(b) Effect of variable splitting
Figure 2: Loss curves of (a) effect of the power p of and (b) comparison between PINNs with VS-PINNs.
with [<| and p- and (ii) the finest splitting of Φ0 R2 u. 01 Vu, Φ2 2 Ugas Ø3 R Urer with |s] = 1 and = 2. In order to train PINNS for fourth-order PDE, p should be at least five, but training such PINNs with GD does not proceed properly, as illustrated in Figure [ Consequently, the experiments were conducted using the Adam optimizer. In contrast to the underperforming PINNs. VS-PINNs are effectively trained even with GD, as illustrated in Figure of the Appendix We run each model five times with different random seeds, and Figure (b) depicts the training loss of PINN and two VS-PINNS with variance. The results show that the model with lower PDE order and smaller power p of the activation exhibits more pronounced reduction in the loss function, in accordance with our theoretical findings. Furthermore, can be observed that the variance of the training loss is significantly smaller for the models with smaller values of k and p. This indicates that the learning process is much more stable for a smaller and p. We also conduct numerical studies on the fourth-order bi-harmonic equation (58)). However, the results exhibit similar trend to that observed in the beam equation and are therefore presented in Appendix E
6 Conclusion
In this paper, we proved that the gradient flow of PINNs converges to a global minimum and provides sufficient width for this convergence. It extends the results in (25) to general PDEs and activation functions and provides even tighter conditions on the width size. The main theorem demonstrates that the PDE order or dimension exponentially increases the width requirement, theoretically indi- cating that PINNs are challenging to optimize for high-order or high-dimensional PDEs. We also substantiate that the PDE order amplifies the adverse effects of dimensionality, which explains why PINNs are more susceptible to dimensionality than other deep learning losses without differentiation. Furthermore, We showed that the variable splitting strategy improves convergence by reducing the differential order included in the training loss function.
It is acknowledged that we only provided sufficient conditions for convergence. To fully comprehend the role of these factors in optimizing PINNs, it is also necessary to establish the necessary conditions linking PDE order, dimension, and convergence. Given that the primary goal of PINNs is to approximate the solution of PDEs, it could also be a limitation that all discussions were limited to empirical losses with fixed collocation points. It would therefore be a worthwhile future direction to analyze the conditions under which the expected loss converges when training collocation points are randomly sampled per epoch. Extending our theoretical framework to analyze the impact of the variable splitting strategy on the generalization eтror of PINNs, as suggested in (64, would also be an interesting and important research direction. Moreover, as our analysis was confined to continuous time flows, a comprehensive understanding of gradient descent would necessitate the analysis of discretized flows, since GF and GD have different dynamics
[50]. We expect that our theory could be adapted to GD dynamics by using Theorem 3.3 of (50), which treats GD as GF with a counter term, but we leave it for future work. In practical context, the convergence of PINNs for adaptive optimizers, such as Adam 35 or L-BEGS and other activation functions, including hyperbolic tangent, remains an open question.
10

---

7 Acknowledgements
This work was supported by the NRF grant (202111//223300008// RS-2024-0006111,,, RS-2024- 00343226, RS RSS-024-004221200 and SSXTHITPNN022021.0011444111rii. Intelligence Graduate School Program(SNU)..
References
[1] Ahmed Abdeljawad and Philipp Grohs. Integral representations of shallow neural network with rectified power unit activation function. Neural Networks, 155:536-5555 -500,, 2022.
[2] Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, Jose Santamaría, Ahmed Shihab Albahri, Bashar Sami Nayyef Al-dabbagh, Mohammed A Fadhel, Mohamed Manoufali, Jinglan Zhang, Ali Al-Timemy, et al. A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. Journal of Big Data, 10(1):46, 2023.
[3] Abdon Atangana and Ernestine Alabaraoye. Solving a system of fractional partial differential equations arising in the model of hiv infection of cd4+ cells and attractor one mee-immeniooal Keller-Segel equations. Advances in Difference Equations, 2013:1 2013.
[4] Roberto Ballarini. The Da Binci-EulerBBeroulli Beam theory? Mechanical Engineering Magazine Online, 2003.
[5] Guy Barles and Halil Mete Soner. Option pricing with transaction costs and a nonlinear black-scholes equation. Finance and Stochastics, 2:369-397, 1998.
[6] Shamsulhaq Basir. Investigating and mitigating failure modes in physics-informed neural networks (PINNs). Communications in Computational Physics, 33(5):1240-13)) 1269, 2023.
[7] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey, Journal of machine learning research, 18(153): 2018.
[8] Feliks Aleksandrovich Berezin and Mikhail Shubin. The Schrödinger Equation, volume 66. Springer Science Business Media, 2012.
[9] Jiang Bo-Nan and CL Chang. Least-squares finite elements for the stokes problem. Computer Methods in Applied Mechanics and Engineering, 78(3):297331, 1990.
[10] Laxman Bokati, Vladik Kreinovich, Joseph Baca, and Natasha Rovelli. Why rectified power (repu) activation functions efficient in deep learning: A theoretical explanation. In Uncer- tainty, Constraints, and Decision Making, pages 7-13. Springer, 2023.
[11] Kurt Izak M. Cabanilla, Rhudaina Z. Mohammad, and Jose Ernie C. Lope. Neural networks with relu powers need less depth. Neural Networks, 172:106073 2024.
[12] John W Cahn and John E Hilliard. Free energy of a nonuniform system. i. interfacial free energy. The Journal of chemical physics, 28(2):258 267, 1958.
[13] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica, 3711))1127-1733 2021.
[14] John Rozier Cannon. The one-dimensional heat equation. Number 23. Cambridge University Press, 1984.
[15] Charles K Chui, Xin Li, and Hrushikesh Narhar Mhaskar. Neural networks for localized approximation. mathematics of computation, 63(208):607-62) 1994.
[16] Bernardo Cockburn and Chi-Wang Shu. The local discontinuous galerkin method for time- dependent convection-diffusion systems. SIAM journal on numerical analysis, 3546)2244--2433, 1998.

---

[17] Peter Constantin. Euler equations, Navier-Stokee equations and turbulence. In Mathematical Foundation of Turbulent Viseous Flows: Lectures given at the CIME Summer School held in Martina Franca, Italy, SEptember 1-5, 2003, pages 1-43. Springer, 2005.
[18) Tim De Ryck, Florent Bonnet, Siddhartha Mishra, and Emmanuel de Bézenac. An operator preconditioning perspective on training in physics-informed machine learning. arXiv preprint 2xXiv23300005001 2023.
[19] Tim De Ryck and Siddhartha Mishra. Error analysis for physies-informed neural networks (pinns) approximating kolmogorov pdes. Advances in Computational Mathematics, 48(6):79, 2022.
[20] Lokenath Debnath and Lokenath Debnath. Nonlinear partial differential equations for scientists and engineers. Springer, 2005.
[21] MWMG Dissanayake and Nhan Phan-Thich Neural-network- based approximations for solving partial differential equations. communications in Numerical Methods in Engineering, 10(3):195- 201, 1994.
[22] E Kirk Donald et al. Optimal control theory: introduction. Mineola, NY: Dover Publications, Inc, 1970.
[23] Nathan Doumèche, Gérard Biau, and Claire Boyer. Convergence and error analysis of pinns. arXiv preprint 2rXiv:2305.01244 2023.
[24] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:/810.02054 2018.
(25] Yihang Gao, Yiqi Gu, and Michael Ng. Gradient descent finds the global optima of two-layer physics-informed neural networks. In International Conference on Machine Learning, pages 10676-10707. PMLR, 2023.
[26] Roland Glowinski, Hao Liu, Shingyu Leung, and Jianliang Qian. A finite element/operator- splitting method for the numerical solution of the two dimensional elliptic Longe-ampdrr equation. Journal of Scientific Computing, 29:1-47,0019
[27] John B Greer, Andrea L Bertozzi, and Guillermo Sapiro. Fourth order partial differential equations on general geometries. Journal of Computational Physics, 216(1):216-246, 2006.
[28] Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 13741, 2021.
[29] QiZhi He, David Barajas-Solame Guzel Tartakovsky, and Alexandre M Tartakovsky. Physics- informed neural networks for multiphysics data assimilation with application to subsurface transport. Advances in Water Resources, 141:103610, 2020.
[30] Masanobu Horie and Naoto Mitsume. Physics-embeddeer neural networks: Graph neural pde solvers with mixed boundary conditions. Advances in Neural Information Processing Systems, 35:23218 22-2221--3322,
23229,2222222 2022.
[31] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 3(5):551-560, 1990.
[32] K Hosseini, Sadri, M Mirzazadeh, YM Chu, A Ahmadian, BA Pansera, and Salahshour, high-order nonlinear schrödinger equation with the weak non-local nonlinearity and its optical solitons. Results in Physics, 23:104035 2021.
[33] Zheyuan Hu, Zekun Shi, George Em Karniadakis, and Kenji Kawaguchi. Hutchinson trace estimation for hign-dimensional and high-order physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 424: 11683,, 2024.
12

---

[34] Deqing Jiang, Justin Sirignano, and Samuel N Cohen. Global convergence of deep galerkin and pinns methods for solving partial differential equations. arXiv preprint arXiv:2305.06000 060000 2023.
[35] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[36] Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358: 12623, 2020.
[37] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 11225548-26600 2021.
[38] Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9():9877 1000, 1998.
[39] Hyuk Lee and In Scok Kang. Neural algorithm for solving differential equationn. Journal of Computational Physics, 91(1):110-14 10-131, 1990.
[40] Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40): 2019.
[41] Xin Li. Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer. Neurocomputing, 12(4):327-343, 1996.
[42] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.0888) 2020.
[43] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-522, 1989.
[44] Minliang Liu, Liang Liang, and Wei Sun. A generic physics-informet neural network-based constitutive model for soft biological tissues. Computer methods in applied mechanics and engineering, 372:1 13402, 2020.
[45] Yu Liu and Wentao Ma. Gradient auxiliary physics-informed neural network for nonlinear biharmonic equation. Engineering Analysis with Boundary Elements, 157:272-282, 2023.
[46] El Mehdi Lotfi, Mehdi Maziane, Khalid Hattaf, and Noura Yousfi. Partial differential equations of an epidemic model with spatial diffusion. International Journal of Partial Differential Equations, 2014:1-6 2014.
[47] Tao Luo and Haizhao Yang. Two-layer neural networks for partial differential equations: Optimization and generalization theory. arXiv preprint arXiv:2006.1577 2020.
[48] Revanth Mattey and Susanta Ghosh. A physics informed neural network for time dime-dependent nonlinear and higher order partial differential equations. arXiv preprint 2XXv:2106607666, 2021.
[49] Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics- informed neural networks for approximating class of inverse problems for pdes. IMA Journal of Numerical Analysis, 42(2):981-1022 2022.
[50] Taiki Miyagawa. Toward equation of motion for deep neural networks: Continuous-time gradient descent and discretization error analysis. Advances Neural Information Processing Systems, 35:37778-3779.1 2022.
13

---

[51] Eike H Müller and Robert Scheichl. Massively parallel solvers for elliptic partial differen- tial equations in numerical weather and climate prediction. Quarterly Journal of the Royal Meteorological Society, (406685):608-26633 2014.
[52] Frithiof I Niordson. On the optimal design of vibrating beam. Quarterly of Applied Mathe- matics, 23(1):47-53, 1965.
[53] Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
[54] Yesom Park, Tackyung Lee, Jooyoung Hahn, and Myungjoo Kang. p-Poisson surface recon- struction in curl-free flow from point clouds. Advances in Neural Information Processing Systems, 2023.
[55] Yesom Park, Chang Hoon Song, Jooyoung Hahn, and Myungjoo Kang. ReSDF: Redistancing implicit surfaces using neural networks. Journal of Computational Physics, 502, 2024.
[56] Yesom Park, Changhoon Song, and Myungjoo Kang. Beyond derivative pathology of pinns: Variable splitting strategy with convergence analysis. arXiv preprint arXiv:2409. 20383, 2024.
[57] Shige Peng. A generalized dynamic programming principle and equation. Stochastics: An International Journal of Probubility and Stochastic Processes, 38(2):119: 19-134, 1992.
bellman
[58] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686 707, 2019.
[59] Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.
[60] Roger Temam. Sovier-Stoked equations: theory and numerical analysis, volume 343. American Mathematical Soc., 2001.
[61] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3355-A3081 2021.
[62] Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kemnel perspective. Journal of Computational Physics, 449:110768, 2022.
[63] Jue Yan and Chi-Wang Shu. Local discontinuous galerkin methods for partial differential equations with higher order derivatives. Journal of Scientific Computing, 17:27-47, 2002.
[64] Marius Zeinhofer, Rami Masri, and Kent-André Mardal. A unified framework for the error analysis of physics-informed neural networks. arXiv preprint arXiv:2311.00559, 2023.
[65] Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using physics-informet neural networks. Science advances, 8(7)xeabk0644, 2022.
14

---

A Mathematical Notations
We introduce the symbols and mathematical notations that frequently used in this paper.
Symbol
Description
No E
The set of non-negative integers
dimension of domain
Ω R
d-dimensional domain multi-index of dimension d order the governing PDE differential operator
a, 3 E No
k e N
N
N
Splitted differential operator boundary condition operator
B
integral partition of [0, source function
f:Ω + R
9: 00
boundary function (augmented)point in Ω
* (
(9)
(augmented)point on a neural network
Gw Go
Gram matrices
Ao
The minimal eigenvalue of G number of the width the network activation function
m EEN
p PEN
power of ReLU activation function the number of collocation points sampled from the domain D, 02 residual of PDE loss
no: Th E
μ
residual of gradient matching loss residual of boundary loss
h
For positive integer m, the set fI, .m] is abbreviated as [m]. The of multi-indexes whose size is m or is at most m is referred by
I (a1,..., No: |a| =
Σ
m
1=1
(18)
Jm (a1, (o1,...,,aa) oa) No. |a| Σ
04
m
In this paper, we use two-layer MLPs. For given & E N and a partition 0 = Eo < E1 < ...
< E1. <
E1T1 define Φe IRd el of width m
Φe (x; we v2)
(ve), (((we)r (we)
(we) x + ((we)
v
(19)
(we), a (wez), 9).
where weights in the first and second layer are
(wz), [wetr, (we)nz (we)dd 3/ (wena+t) g(d+1)
(20)
(vz), (we)r, (we).. (wr),, E Ellsel,
(21)
and y [m= } Rd+1, Note that the output dimension of tee is R Aeel because each component ašz
of de (æ; we, we) represents partial derivative 3 u, for each E Te.- Hence, even though (wz),
flattened vector, more convenient to use a Te. as an index of component (æe),- For
example, if d = 2, 1, and 10, I, (Vi).a (v1)r.0,1) refers weight between the
15

---

r-th hidden node and output that represents @ (z, y). Similarly, we use multi index itself as an index that is related to & Taq. or Jac.
d for a fe--: and E Lace> index we Moreover, as each ( differentiated by some an g
pair of multi-index () to refer component of da (Øt)ra Sometimes, this indexing by multi-index is used in conjunction with normal indexes, like (Me) (a. for some i [no].
We denote the collection of weights of all de's by w [w]] w wI] and [ "[]".
Similarly, 1l is the collection of all py s, μ = [MT. ... _u]] With regularization parameters v, V 0, residuals Sis (1z) i, and h of variable splitting for each training sample
(w,v) = (N 1d0 (:3 wo vo), , (: vL) (z:))
(22)
(142); (w,v) = [(e)) (w, u)] OETE,
(23)
(0t-1)a (x we-1, ve-1) oe (x) we, vel aE BETD
(24)
and
V"
V' (B (: wo. wo)l (tt) (i,))
hj (w,
(25)
nb
define the loss LYINN for the system,
LVS PINN
(w. v))
(26)
no
Ro
18
Σ S W. 0)²
Σ
Σ
(Pe)aa. (w, u)²
Σ j=1
hy (w, v)²
(27)
oE BEJaet
we define
Gmax = max (f ((iecas (ax)l:
(xx)): a E Iec 3 Joee
0 L,iE [no],
} u Iāa (ă,)1:
j je [np]
a Jr,
}).
(28)
Viaxx =
max (pl [4]))
(29)
B Calculations
The proof the main theorem includes intricate calculations. To keep the proof clear, we separate some tedious computations that used frequently. As this section is reference for the proofs, use some notation or symbols without any mention if they are defined in the other part of the paper.
Sizes of index set Im and Jm-
For any le [L].
(d + Ee -11 |TSSal= (d+ ACe - 1)) .JASS.
(d+ A&E) )
1Tg.
(30)
Ee
A
D
We bound [1g. [AEll and JAs by
dd+1l)
Las S |Jasel S
(dd d
(31)
and
f.
Us Jss 11 ( (d+h)
Σ
(32)
2=
16

---

go Partial derivative of 8, pl, and h. Note that Jz
(wwe) yi
in 34
(wee-1), Vi P
in (35). (wwe), yi in (3)) and (we),, and (wo), of degree p. respectively.
8
((wo), yg in 37 are polynomial of (we),- (we-1)r-
For i E [nol, l = 0, ,,...,,,... L. E [m], and a E Lee
I E
08 0
(U:)r.
81
no BEJaEr-1
a 8 Oxp a (vs)r.a
de (2)i)
(33)
08 (1wi)) Vi)
Σ
a Dx
(34)
mno
BEJaCs+
Similarly, for e [4], r E [m], an 1 02 B5 Isce and i [nol, we have
i
8
((we-1) v.) ))
if af 02 otherwise.
(He)
mnoo
(35)
a (vz-1)r,
For l [L], E [m], an lee as Lee 1 E Isce and E [moo]
(wwe), w1) P
Vr mna
if = 02 + 3, otherwise.
(1e)
(36)
① (vz)raan
For r E [m] and j [mb).
v
ga ((w))) )
hj
aa ax
(37)
i (vo),.1
menb
aE
Norms of partial derivative of s, μl, and h.
Note that for i E [nol, l = 0, 1,.,,..,,, L, r [m].
Os; (w, v)
8 (wi)) ((ws)) 3)
Σ
i
(w)),
2
=
mn,
Ō (ww),
aeTe, BEJase+1
a
Fx
(38)
1
83 (ue)rra (<wwz)) 3
)2
ar da.
mn.
b (w))r
arlle BEJace+1
(39)
Σ
(ve)ra
Σ
0 (we),
g3 (ww1) 3)
P
mn
aElez
BEJach+1
(40)
08
(wwz) 8) ))²"\|
mn,
Omax
Σ
(ve)aa
a (we))r
(41)
a
BEJOEEI3
Qmax
Σ
(Delraa
Σ
pacetit] ((ws),11
(42)
mno
acter
BEJOEE+
Omax
(De)r.a
|Jssmi
p (1
(43)
mno
aETEr
5
Ounax V M (eelll(ee) |Jscm /p [1(ws),
(44)
mnno
17

---

for l E [L], E [m]] a 1 8 E Isc and i E [nol]
d (a,)), ww 0 (1e-1))
i
(V mmo dxo
Ve
08
(wV--))aa (wws-1)) vi
1 )=))
(45)
2 (w)-1))
Vt V mno
(De-1))aa)
2o ((1oL1)) 31 )
a (w0=-1)
(46)
Vi mna
(ve (ve-1)).a p p 1((0)-1),112
(47)
Umax
p11l+1 11(06-1)r11211 (10
(48)
mno
① (o.0)) (w. w) 0 O(we),
a ① (we)),
Vt ( (V
(wii)na (wwe), yi) (=))
(49)
mna
Ut
(ve)r, 0 (weer
(Iwee), Vi 1 +
(50)
mno
Vi mno
P (w (((x))))12"
(51)
Vmax
p (1((0)))1221110) (1(wwe)
(52)
mno
1 (Hee; (w, v) 3 (wwr) )
= 0, if # 18---4 1,,,
(53)
and for r E [m]] and j E [no].
0h; (w
Oa
(oo)). ((roo), 33 P
āa ax
(54)
i (wo)r
2
a (wo),
mns
TETi
8r
(vo),. ((we)) 8, p
mmnc
amax
2 (wo),
a
(55)
aEd;
Vv
amax
Σ p² (vo)r.o) I1(20)rr11
(56)
v mmz
NEJz
Vmmax
amax
(d - p² (((o))rl2 11(200),
(57)
mns
Thg (w, v)
= Of [L).
(58)
a (ww) )
12
Similarly, for l E [L], r E [m], and i E [noll we have the following inequalities: for a E Ie..
Us: (w, v)
a
T (wwz),
p y. +
(59)
0 (W)) na
mnm BEJaSe!1
Dax
amax | deett (((e)))l1g)
(60)
V
Us; (wr, v)
Vmax
mnn amax / |tsal
|JASEn [
(61)
i (vz),
2
for an E Ie 11 22 E Tee and 13 E JsEe
(w (t),
/ Ve
83 ((we-1)² v1)"
i (ve-1)mm
1 (0s ma) V mno
dx
(62)
18

---

V Vi past (1wwe-1,,13 mns
$ 1 [m] =0))
(63)
1 (14 (a2 (a2)) (w, v)
Vi م v |Ese ./aaeel 11 mno
(64)
i (
for 01 Ieo 02 I and B E IDEE (ag, (w, v) a (Ve)raa
Vi ((wi)) Vi)
1 [a)=act))
(65)
milpp
S 1 [a)=ast9)
Vi
11(w=),
(66)
mmm
(1t) (an)) (w, v) d (vee),
Vt mno
v |5 |uac
((
(67)
2
for E' + 8 - 1,
e (prr) (w, v)
0.
(68)
θ (vvr),
and
Thg (w, v) 0 (vou),.
g
((wn)) 03) )=)
da ax
(69)
mib
aEJ,
Vmax
(d + 1) O (wo),
(70)
mn,
Th, (w, v) Ö (vvr,
0 [L].
(71)
Hoeffding's inequalities for Si,rs (pe)r, () and h3 For each [mo] and [m], let
8
8; (w,
ΣΣ
a a dx (de) f (zx))
(72)
no
8=0 aETee BEJ
_ (x:),
Sir (w,v)
(73)
Tho
r=1
where
20 (weina (wwe)) y) ))())
Si.r v)
Σ /mn aETe, BEJAC
a 8x
(74)
As Str is a p-th degree polynomial of (We),,, we have
88 (ve)r,a (lure), T y) ))()
|8h, (w, w) <
Omax
ΣΣ
(75)
/ /mno E=0 aele, BEJaCH+1
dry
< Fmnno 1=0 BEJaGet1
Omax
ΣΣ
Σ
(De)ma A [(ws))
(76)
for all i E [mo] and E [m].
Since [[(,,,, (0)ll, < R and (werr.a (0) < for all f= f=0,.. O,...,, L, a E Ie., and r E [m], we have
8g. (w (0), (0)) S
mno amax ΣΣ Σ
p A Re
(77)
t=0 aElg, BEJasett
19

---

L
Pmax
Umax mno
1lell 1--ssss -p
(78)
t=
Vmax
[d k) d
(d+ (d (d
Omax
(79)
mnoo
d
for all E [no] and r E [m]]
Using E (we)r.
= 0, we attain EB [st. (w (0), (0))] O, and Hoeffding's inequality gives
m
2
I
sir
(w (0), 2
e
2exp
Vno
Mmax (2
Omax mny
(+tt) (d++
pl 1
(80)
82 2/ 2 (dt 2 (at)el) 1a1el 122
2exp
(81)
for all >> 0.
Similarly, for each / [L], a E 1e--1. Ioee and i [no].
00
(He)(a,)), (w, 0)
dos (de-1)a (an we-1, ve-1) de (er;; we, ve)a48
(82)
no
(pe) (w,v).
(83)
where
G3 (122-))2a ((we-1)..1)) (ve)raat ((we), )1)²²
(14) (w,
Vi
V
mn.
0x
+
(84)
Then, we attain that (pe)). (w, w)
Ve
(1
(we-s)no (wwe-1)) v. ).) + (ve) (lw0e), yo
s mno
d
(85)
Ve < V
(pate (ve-))ra (1(00-)),11 (v0)) ( [((ws),1)).
(86)
vin.
and
(He)) (w (0), ,0)00000 (0))
< / mn.
(p161+1) R.
s
(87)
E (wx) n. = 0 implies E [Me)) = 0 and for each E [na]. Hoeffding's inequality with the
sum over r [m]] of variables
Xr, (w (0),
Σ
Σ
(Me)n (w (0), (0)),
(88)
aETE._1 BETase
with
1x (wo (0)) < TTe .1 |/Eecl V
Vmaxx
(plel) + +1) R
(89)
mna
Unnax
dd k.
cd [5l) (p151 +1) Rd d
(90)
mnoo
d
20

---

gives
E
P
(ee) (0), (0))
(91)
no
2
c
V
S2 exp
(92)
Kinax mi7
(d+ ( (a+tls) (pls) - 1) RP
E²
<2 exp
(93)
((ttt) ( (utlel) (plE) + 1)² R220)
Umax
For each j E [no], let us denote
h (w, w)
Σ
0o (Z vo) - -V _
g
(94)
Thb
nb
aeJ
hir ((wa), (wo)r)
V g (āt))
(95)
mb
where
ga da ox
(1on)..0 (mwn)) ū3 )))
hj.r (w0, v)
(96)
mns
aε
is clear that hir is expressed as a p-th order polynomial of (wo), and we can deduce that
Vmax
1/j,r (w (0), (0)) S
mmax (d + 1) pR?"
(97)
V
mnn,
for all j [mg] and E [m]] Similar to the case ssr and (Hr)), Hoeffding's inequality gives
(-2
E² 2 22 2unx (d+ 1)² p2 12
Σ hjr (w (0), 2 (0))
E
S 2xxp
(98)
VTb
Components of Qiats
Each component Qi,ss is polynomial of order p as follows:
(Qo,0)
(99)
i
8
i (vtraa
Si
Sis
(100)
8= I oE
g0 (wwi) Vi )i))
8
ΣΣΣ
Σ
((wz)) Vir)
0e dx
mno Erl=
BEJaCh+
BEJaer+
(101)
(Qo,) (Qo.) ((a(())))))))
(102)
ΣΣ1
Sis
i
(Me)
(103)
((v)) ) rra
a (
r=0 a'Eler
Σ
m
Σ
8b1
a
(104)
2 (w
(1(-1)
r'
21

---

m
a
Σ
i (ve)nao
Sis
i (vi)ra
(He)
(105)
Ells
m
i
Σ
Σ
Si1
(106)
[1) 1)aa'
2 (V1-1) 1)na
a'E
17
(ve)
811
(ve))nat
(Me)
(107)
r=1
VUV
m
Σ
g0 (we-1)) yis
Ox
0
(wwe-)) yiu) )())a)
+
(108)
mno r=1
08' ((we)) Yis )
((w0)), Vis
Ug Ox
(109)
g'EJaset1
L m
)
a (ve))ra hg
(Qo.14).s
8i
(110)
b (velra
ł=
aEls,
i (va)r.1 8i
a (vu)r1
N1)
(111)
Σ
2 Ox (mw)), v.)²}
03
((> laEd
Σ
da 2x
ga
((wo)) 83 )
Ro
BEJAC
(112)
(Qit) ((an (22.)))
(113)
m
ΣΣΣ
(pL) (Pe)
(an,
(114)
(vv))
2 (ve)ra
0-=0r= aEE
(Me)
(me) (ae)
(115)
2 (v 1)na
r=1 aélir-
0 (vve).
à a (vi)aa
(Me)
(Me)
(116)
aElse
00: Vi
(ww2-1), vi))") (V
Oo mn. 8x
((w(-))) Viz ):)) 1 (a1=a2)
(117)
mn.
8
Vt.
(we)) Vin ).) (-
Vs. mn,
(wwe), Wio via)")) 1 ((11t))
(118)
mm,
r=l
m
851
(wwz-)) 3i.)) ):)
0
((we-17 Wix) 1 1|a
(119)
mn,
(we)) 31. )² ((we), vr.) p 1 fan
(120)
(121)
22

---

m
2 (vp)). (me) (Ho) ) (π(ee) (met2)) (as
(122)
#= Latte
(He) )
a (ve)na (o
(123)
Σ
m
VE
((wi)), wi.) ).) 1(a=aith))
(V ge ((wz)) viz) 1 (a=aa)
mna
aElse
(124)
Veeee+
Σ
(-((11)) ((11)))) Mi.) )r) ( 0x
(mws) Mis) (1on),)))')))) 1 (a2=ar+8.1
(125)
mno
(Qt,Lt1) (1.8))0)
(126)
4
ΣΣ Σ
(Melionoo
a ((ve))aar hg
(127)
a (vv)),
8=0 rml
a Else
a
δ h3 (wwo), vi) ):) (v
(pe)
(128)
a (vo)r.1
8 (wo)r.1
V1
aa
Σ ān (two)) 0))²² [(==1)
(129)
mn,
Oad
mib
aeV,
V1V
Σ
m
00
((wo)) )
Σ
ãa
go
((wi), 81) 1(==13-
(130)
m V no
0x
dx
r=1
LOEJ
Q (Q1-1,44))sue jint
(131)
1
( a (ve)ra (0(o2)aa hjj
(132)
a(ve)na
ł= r= aET
2 (D(c ((vo),.
hij
hiz
(133)
a (vo)r.
ga
((DD)), 8íj )
(V
Σ dn g
(mwo)) أرز )²)
Σ aEJi
da Ox
(134)
mng
mns
Ox
aEji
m
0n
(wwb) T Fú )²
g ((wo)) T 13 83z)) (2)
āa dx
āa
(135)
mmb
ax
LnEJ
Laeh
The number of possible nonzero elements in Q and G. There are possibly ng.
Ie L ng. and n, nonzero elements Qa Q and Q0 respectively. For each (ar, E Ie-- Lase there are at most [ pairs of (a2. 32) E I Iag, such that 0n + 91 = 02 + f2 and at most Taeel pairs of (a2,92) E J x Tae such that on 02- Thus, there are at most 2 Tagel pairs of (az, (02.12) E TT.- Tas with on + B1 02 + A2 or
0n 02, and Qr has possibly (T al | n.) (2|/Agr mo) = 2 Te. elements. Similarly, each 81) E Ie-n x Tag, uniquely determines 02 = a1 + 31 E Te. there are / pairs of (2) E le, x LASci with 02 ai + 31. Hence, Q has Lecall Laeel- | n3 nonzero elements. Q1 has Mas nons nonzero elements. The number of nonzero elements in Q.+1.411 is ng. After all, the number of nonzero elements in G, is
|Taee 2 ng nonzero
23

---

Matrix
The number of nonzero elements
Qo.0
2
Qo.r
I
m
0
nob
Q
2
|sss, 12 ni
Q 1,t+1
[Ec-1
| n2
Q 11,++1
| mig
Q L+1.1+1
Table 1: The number of nonzero elements in blocks in G.
most N1 7(*tt)) (1) ni + ( + ng. because
the number of nonzero elements of G.)
(136)
L+1
the number of nonzero elements in Qr,e)
(137)
t=
2
(the number of nonzero elements in Q(,e)
(138)
#=t+1
<n +
2 Te |TArr1 n²) + n
(139)
+
Σ
(1 | nz) + no
(1 | Lacen n.) + Hoae nonl
(140)
E=1
dd (d+
+51
2 L
Sng + 2
Σ
|g n + i
(141)
d
t=1
(d+1el)
(d+1k)
(d [E]
+2
Σ
LEe [/o noms +
Σ {=1
Iec m +
notlb
d
d
t=1
(142)
1 2
d dd+1 [4)
d )
(dd d
dd+ /d+
+ 4
dd+ d+)
d /d+k) k)
))
n +
1+
dd+ (d+1g))
no
nib 2
d
d
(143)
dd* k)
d +
2 2 4
(d (d+
n + n?
(144)
d
-Ni
(145)
Norms of partial derivative of Qi
For each 11, E [mol, r E [m], and & e {0} U [4]. we have
(Qo.0) 11.22
(146)
8 a das
8 T yis)) )2
Σ
(ww2), T Vi )²
ae.0.3 dx (wez)
mn. E=
nETEr
BEJaset1
BEJase+1
(147)
i
(Q.00
(148)
(wrr)
2
24

---

1
Σ
Σ (8EJSE++1
8 ((wvz), T. Yo.1 +
)
08 af Ox.
(1wi))
yks )
ag dx
mno a (wee)
afder
BEJACI+1
+
(149)
|/ael mmax
a 2 (we))
88
((wi)) Vis) Bir) )²²
Σ BEJDeets
0
((we)), yiss
P
dxe
dos
BEJAEE-11
(150)
max
ITel Jasetal I2 12111+1 (1(ws), 112p-1
(151)
mna
ô (ve), (Qo.u) 12:82 = 0.
(152)
For each E [1], E TE..1. B Ta and 12 E [mo]
(9.t))
(153)
VE
Σ BrEJasr
ât- 10.8
08"
(1 vis) )
00
((we--))+ wis) )
(154)
0x
dax
mna p=1
3 Oar (wws)) win) w.)).) ((wi)), Vis ) P
g8'
(155)
B'EJaCe+1
0 [w ((WF-1), 1)
(Qo,)) (t.((a (1.
(156)
2
Vue
Σ B'EJASr
0a (mne-)) ) )
( dxs
((12-1)7 Viz)
11-10. Darl
mn, δ (w
2
(157)
VEE
Omax
Jage 1p 2250t1 1(1re-1), 1120-1
(158)
mng
/Vmax
(a1 p 2111+1 (1(6-1),1122 20-1
(d+ (d+11))
Omax
(159)
mn.
and
0 1 (w)z),
(Q..)
(160)
Vvi
g daror
((wi)) vnn) ).) ((wo)) Vzz)
P
0
(161)
minao ① (we),
B'EJAGE
Vt
Omux
Jasee
p D ([(D)) 122011
(162)
mno
VVmax mno
amax
(d+1El) d
p 12161411111 [1(0s), 2p-1
(163)
0 a (wr),
=0 if * t 1,
(164)
(00.e)...(la.))).))) ((a,)),))) = O. ve E [7]] 7 (ve),
(165)
For each i E [no] and [no].
(Qo,4+1)ss
(166)
25

---

1
m
Σ
1.
((wo)), 31)²²)
+
Σ arEh
an
8a an
((wD)) 81
51)) :)
(167)
m
dx
BEJAC
0
(Qo.t.t)as))
(168)
(wo)
1
i no d (woor
Σ
1 dx
8 (wwD)), 8. ))
Σ NaEJ
do Ox.
ga ((wo)) B)) P
m
BEJaC
(169)
1
d A61)
(d (d+ 1) a maxt A6it2 [1(00)), 122-1 1
(170)
m
no
d
Vimax
d [sl
d ) (d + 1) maxx pl1++ [(200),112 10-11
(171)
V nomd
m
0
0 (w)), (Qo.L+1),, i.j 0, J4 00
(172)
Qi,j 0, Vl.
(173)
(ve),
For each E [L]] a2 ag E Ie-. Sr, 32 E Lsg and i1,22 E [no].
(Qu,) (Gnn
(174)
Vt
Om (wo Vi:) )t)
00
da ( (wwr-1) )² 1 [ai=as)
go:
(175)
mno
r=1
p + (wwz), Vti) ) ((wz)) Uia
1 ((1+t12astt1))
(176)
a
(177)
0 (w
2
Vi -1 [a)=as)
{
((00)
g0:
(( Vi.) (1.1=)))''1'" )o)
(
dx ((wz))) vv.)
0os
mn, d (Wi- 1)p
(178)
<1 (a) [a)=ag) mno
Vi 3
p
[att+I[[===- (((e)1))l1 201
(179)
<1 (01=00) mn,
Vmax
p 211|+1 [1( 21201
(180)
and
(Qe,e) ((an ((nn)))3))(((00000)).))) 2
(181)
δ (w)),
Vt σ mn, θ (we),
{ (wwe)) Ue.) [(ww), Uia) )²,)
-1 (01401102222)
(182)
<1 =a
Vi p (w=), 2p-1
111
(183)
mnn
Vmax
[|( (ww)),l1 21p-1
(184)
mnn
(Q (,.2 = 0, 4E' + l - 1.
(185)
a (we), )r
i (Qt,)) = 00 VL. 0 (ve),
(186)
in:12
26

---

Similarly, for each e E fl,... L - 1}, as E Ie-- az E Iee Br E Iscrsp2 E Isem and f i2 E [nol. (9 (1o)),)))ii)(((zz.aaa)))
(187)
yveve+1
Σ
- ((wx))) Vi.) ).)
(wwz), Vis) vi)).) 1 102=a1+
(188)
minoo
(189)
a ① (wi),
(Qe,4t)) (10441/((000))))))00000000..
(190)
VEEVE+I p paleistl ((w=), 112 22p-1
<1 (12=
(191)
mnt
Vmmax kimxx1s1+t1 pllt+1 11(e),,112 120-1
51 {aaz=ai +91
(192)
mnn
0
(QE,41) ((1))))) 0, VE' te,
(193)
b (
(194)
0 (
(Q.,4+1) ((a.)))))))( = 0, AR'.
(195)
For each E [4], = 1, E Lsc. i E [mo] and j E [ma].
(196)
1
VIv
Σ
m
88
(wo)). vi) )
Σ net
āo Ox
ga
(lwo). v.).) (..))..i)")
((-=)).
(197)
Ozd
m
p=l
θ
(198)
a (wo),
my (wo)r
V
m
3
2
((w0)) v.) )
Σ
Za dx
8
((wo)) T 8, +
(199)
V1DB
mo Omax (d + 1) p ((wo), 22-1
(200)
m maax (d + 1) 11pp (wo)) 11 (2p-1
Vinx
(201)
a (we), (QI, ((a,)))) = 0, Ve 00,
(202)
7 (we)r (Q1,L41) (la,)))s = 0, AE.
(203)
For each J1. E [mol. the direct calculations lead to
(QL4).4t))nnaa
âo
8 Dar
((wD)) 83 ).)
Σ
ān
ga
(wnn), 01:) )
mmg
LaEJ
Ozr
+
(204)
a
(Q
(d+1)2 5 _ 2 (d 1)2 p² (1(wo),)122 218-1
(205)
0 (wo),
0
(Q
2152 = 0, Ve # 0,
(206)
i (wr),
0 d (vt),
(QL+I.L4))
51.52 =0, Ve.
(207)
27

---

Hoeffding's inequalities for Q
For each E [no]] and j E [nol, we have
(we)
)
$ splo ((we), 12
(208)
3
+
00
(we-1)) Vo)) )
$ pl ((we-1),,1.
(209)
ix
((wz)) v.)
$ (1
(210)
ga
(www) _. 83) )
S spld aal ((wo)
(211)
2x
Then, for each i,,12 E [mo]:
Σ
Σ
0 dx
0 (mwe)) vin)
Σ BEJace+1
80
((nne)+ Uis Vin) P
t=0 aElee
BEJARr+I
(212)
Σ BEJaEE11
amaxp
191 [(
amaxp
181 [(
(213)
8=0 aElee
SEJsEe11
JASK 2 amaxP"
21 /
R
(214)
1=0 NETEE
1
+ 15)
Heel
amasep 2|4| 1221 R2D
(215)
d
Ł=
d +
dd+ + [1) k) d
2
d <maxp22l 21 R2D
(216)
By Hoeffding's inequality with
(Qo,)) i2.2 m
(217)
ΣΣΣ aEla,
a (verra
sin
a b (ve)ra
sig
(218)
m
8 ((w)) Vir) )²
(wwz),
T
at Oa
a Iad
Vis
mn.
t=0 r=1 aElee
BEJacr+1
BEJacett
(219)
have
P P[((uu)) 11.22
E [Q0..) [2,52] > < 2exp 2(2tt) (2tlel)
mn?e²
(220)
D 1848
for any e >>0 O.
Similarly, for each & E [T], a E Tec 3 E AEc, and in, is E [mo].
Σ
0r
(wwe-1), v.) ).
08
(wwe-)) Bisa )
(221)
dxs
B'EJAse
8
(wi)) Vis) )x) (wws)) wts) p
Σ
(222)
g'EJacet1
28

---

d D AEc
d Afe+1
amaxP As: 11(we-1), p/ (()
amaxp Aeel1
(((we), 112
(we)),
Actz1
(223)
d + (41) d
(pIs) +1) dmuxp R
(224)
By Hoeffding's inequality with
(225)
VVV mno
m
Σ
00 (1we-1)7 Dis) )²
( dx
008 ((we--)) Vio) )i))a)
â L
(226)
r=1
88' ((we)) Vis) )in)t) ((woe), : Uio Bis)
og dx
(227)
8'EJate11
have
P (Q.s))
E
[(a..a..t >E < 2exp
mm
2ve (ateee 2 (ple)
1)² a mm D21E RAP
(228)
for any & > O.
For each i E [mo] and j E [mb].
Σ
1,3
a
(wo)) v1)²"
Σ
da 8a
((1D)) U5) +
(229)
Oas
BEJACI
d
AEI
d dd1 amaxp ASi (1(w0)).11
Omaxp [(w00),
(230)
ACI
1
d 15)
(d 1)
2 ml |s++1 RpP
(231)
By Hoeffding's inequality with
(Qo,tti)ss
V
1..1.3 Das
0 (woo) vi) vi)²)
Σ lacn
Za gr
((w0)) 8 )²²
To
Ox
AE AEJAE
(232)
we have
(QQ.Ltt) - [[(0.L--))s]] > < <eexp (-2v 2v (d 1)² (dt1e)
216 Rdpp
(233)
for any 8 >>0 0.
For each E [L], an, a2 Iec 11,332 E Jase and ii, E [no]]
( Onn
g0r (000 ((WL-1)) yi )²)
De (wwe-1) yiz) )/ 1 1ai=as)
002
(234)
+ ((we), Vin)² ((we), Vix) 1 ((11
(235)
< p1811+1@11 R2
(a1=as) ( +
R =02+2)=
(236)
< (p21el +1) R²P
(237)
By Hoeffding's inequality with (Que) (()..))))))))oo))))
(238)
29

---

Ve
Σ
((
06: 0z (110--1), Wis )²) ( 0
dorl (1wx-1)7 Vis) vin)) )1)))) 1 (a1=as)
(239)
mna
r=1
+ (we)) vis) ((wre), Vis) 1 (01+01=22t()))
(240)
we have
mn3e²
P (Le( (con) E (1n)) >0] 8) < <2exp
21 (pal6 1)² 1848
(241)
for any & >>0
For each & E [4], an E Tee 1 02 E le 31 E TDEEr B2 E TeEe and i1, i2 E [nt]]
((we). Mi.)) )² ( 00:
((ws)) Viz) )in)a) 1 (0g=a1+84) S 515) R20
(242)
0z
By Hoeffding's inequality with (Q 1) (lon ((o)).))..d)))(((ro02......
(243)
V Σ (- ((w)), )i)))) 8x
g0;
(wwz) Vin) (112)_B1))")) 1 (002
(244)
mn
we have
mage²
P (Qeet)) (oo - E
>]] < 2exp
22eue+1pPle Rtr (245)
for any E > O.
For each a Iee 1s 3 E Joser E [no]] and j [np).
(lwo)) ) a
a
P
Σ
Co'
dxa ((10)) BB))
go'
)))² +
(246)
I p1 Rp Σ
mmaxp al'l
RPP
(247)
o'EJ1
< (d + 1) (11x [e|+1
1p
(248)
By Hoeffding's 's inequality with (Q1,t+))
((a,8))5)s)
(249)
V1V
(www), vi) ).)
(
Σ
go' dxa
((w))) 8 8)))²
Ca
(250)
m
To
Sx
La'
we have
P (1.L4))
E. [[QLLt))aaoooo e]] 2exp
MnoNGE
2 u 1)2 02 211122 1
(251)
for any 8 >
For J1,J2 E [mo].
Σ da ax
g
(mm0)) (ono))/,)))) ).)
Σ
ga (mwo). (iin) 31)) < (d+ (dd11) 1)2a2 2 RPD. (252)
da Fx
aEJi
LaE
By Hoeffding's inequality with
(QL+1.142)) 3:
Σ
da aa
g
(wo)) 8,,) (mbo),,,))² ).,)%²)
0r Ja dx LEE)
(wwD), "oo0),")))²²
mng
+
aEEJ
(253)
30

---

we have
mage 2v² (d + 1) m RAp
P (9 ( (QL+1.4t) s1.ta 2eep
(254)
for any & >
C Proofs
This section proves the main theorem with some lemmas and propositions. Similar to the [25]. we first prove that the Gram matrix G, is independent of m and strictly positive definite, in Proposition
C33
Second, as G (w (0) (0)) is the sample mean, Proposition C.4 shows that G, (w (0) v (0))
is close to its expected value G, with a high probability, if m is large enough. This implies that G. (w (0) (0)) is strictly positive definite with high probability. Then, show that the initial loss bounded with a high probability if m large. This is because v (0) has zero mean, which hinders the output of variables diverge
increases. Finally, Go (w (t) (t)) remains within small neighborhood of the initial point, where it maintains its strict positive definiteness. The gradient flow converges within the neighborhood, as the smallest eigenvalue of Go is distinct from zero, resulting in rapid reduction of the loss.
prove that the Gram matrix
Lemma C.I. Let d, n N and y1..... Vn E Idd vectors such that 11ll y/ if and only if j. Then, for each E [n], there exists w; such that w y = O if and only i = =j.
Proof. Let y f E Rd w y, = 0}. Since y, y for i * j. y, 0 y is nowhere dense in y Hence, finite union of U nowhere dense in y and in particular, there exists w, y, such that W; & U
Lemma C.2. Let n, N, and R.d T.. be homogeneous polynomial of degree For any i and nonzero y = (y2...., yd) E Rd, a function ib (w) (w y) of w = (W1, wa) E Rd (n + iimie continuously differentiable at some w E Rd with (w*)² y = O. then 3b =
Proof. Without loss of the generality, may assume the yn + O. Write to (w) EoEl, ag w
for coefficients 0, R and define yp (wn (w)) Ty) For any a I,. the assumption differentiability implies that 8' 8"
Owj awn 1p continuous. Indeed, we have sp (w) for w y S 0, and
0 ar
dw Owa ip (w) ala.vi.
(255)
for w y > 0. The continuity induces aa = and thus ib = 0 for all w E TR..
Proposition C.3. Go0 = Ew. Go (w, v)] is strictly positive definite and independent of m.
Proof. For forin, i2 E [no].
(Qo.) inviz m m
(256)
8 (wwe) vi))
8 ((wee) yis
Σ BEJacH+
a dz
Σ BEJaen
0g dx
no
aEle,
(257)
and m random variables
ΣΣ '
Σ
Σ BEJace+
8 ag ar (wwe) v..)
Σ BEJAcr+
((we)) _ yis)) )is)""
de ax
no 1=
(258)
31

---

for r [ml, are independent and identically distributed. Therefore, the expected value of each component of Qn, independent of m, as the expected value of the sample mean of independent and identically distributed random variables is independent of the number of samples. Similarly, each component of G. (w, 2) is the sample mean of independent and identically distributed random variables, and its expected value Go is independent of m. To show that Go 00 is strictly positive
definite, assume that there exists zo = [(z0) and
(zo)m.) E RR == 121
]nd] E IR"",
2, = (a,))) (a (elae,
SElss, dElmal) E Rl'se MH
(259)
for 8 [L], such that = [ z
= TJ] E ERno+tia
[ (mo) satisfies
G, 2 = O. Since G. 2 = Fw,y
Do z = 0 if and only if Eu
2 || Doz
= 0, we show
2
that E
[|Daz
= O implies = = 0 and thereby is not an eigenvalue of G.
2
Now assume that there exists E Rd such that D.= = 0 for almost all w and v. Since D2 is continuous with respect to w and v. we have D. = 0 for all w and v. Hence, the function D, of w and v, which is identically zero, is smooth with respect to and v.
In this proof, denote an index for component of D,, by E [4], r E [m]] and a E I because each row of D. corresponds to the partial derivatives with respect to (we)r,a for l E [4], r E [m]]
and a E Ig,, For instance, (Doz) 0,1.0
is the first component of (D.z). First, for = L and any
r E [m] and a E Igz. we have
(260)
L,ra
Ro
i (VL)na
0
81 8)
(zo);
+
Σ
Σ
Σ
① (vL))a
( (p))aor
(zL) ,1.
1=1
E'EEE--1
AE BETTRL
(261)
Σ
aL dx
88
((wL), Bi)) (zo);
(262)
mno
DEJaEL+1
na
VI.
Σ (WL)) )1) (EL) 0), (a=art)))
(263)
mn.
BETacZ
Fix i [no], and by Lemma C.I there exists (wi)), Rd such that (wit), y, =0, (w))), y. + 0 for # i' E [no] and (wi)), ú; + 0 for j E [ns]. As a function of (WL),, D. = 0 implies that
28 (ww)) ".)", (zo):
Σ
(264)
BEJAEL+1
(wi)) 31)x, (2 la=a(+9)
VV..
(265)
d'Eler-1 PElAGI
8 a ord (wwr)) v.).) (zo);
(266)
BEJAEL+I
(WIL) )1) (EL)(a) ,9,, (a=a'+9)-
+ V1.
(267)
V BElacr
32

---

is smooth at (WL), (wi), Note that the coefficient of ((wL), v:)
)
n n--EE+11
in 266 is given by
LL,a.3 (x:) (wL)); (WI) (zo)s
(268)
which is homogeneous polynomial of u. Together with the fact that at least one of 3 (x:) is nonzero, Lemma
C.2 implies (zo); = 0. Similarly, for any r E [mm].
(D0z)
726
a
hj
3
(269)
0...0
a (vo),,0
The
80
((wo) 85 2g
da (t:
(270)
mn
nr
j=1 aEd
aa (2,) 2j (wn)..i ... (wo) (ww)), 81 p-1ol
(271)
m&
aEJ1
deduces 2; = with wy. instead of wi. such that (w)) T ú, 0 0. (w)) T Dy += for + j. and
(w)) T y; f 0 for i [no] Now suppose that we have ă 0 and zę 0 for l' 0, 1,.,... 1
for some 2 1. Then, for any E [m]] and a E lee
(
(272)
8 1,,,a
n2
0
no
(zo):
(Me)
(ze)
(1
1)r,a
a a'EJse-1
BElase
(273)
no
Σ
Σ
(He_2) (ar, (ze-1)
(274)
i (ve-
a'Else _Elase-
Ba
@(V(-1)na (Ho)(ar (ze)
(275)
a'Elee-1
BETAEe
Vi
no
... (wel ((we), wi)t )-A6)) (ze) (z))
(276)
BETAEE
Then the coefficient of ((w)) v.) v1)= p--Asr is homogeneous polynomial
84 (2)) (we)
(277)
no pelac,
Lemma C.2 shows (ze) (a,0), = 0 for all B. The induction concludes z = 0.
In the sequel, Ao Amin (G,) > 0 is the minimal eigenvalue of G.. Furthermore, we assume the occurrence of the following event: there exists R > 1 such that [I(we), (0)ll2 < R and [[(ve, (0)ll, R for all ( = 0, ... L and [m]. This assumption is employed solety for the purpose simplifying propositions and their respective proofs. In the main theorem and its proof, we use the following arguments with regard the conditional probability of the aforementioned event.
Proposition C.4. Let 8 > 0 and
d.
d
[5)
1
1 111
v + 1 (p p215+)).
+1),
G
k
[E1
22
Vmxx
noo
+
12b
(278)
+ k)
+ [E)
(d
N1
n
+
non + 2
(279)
d d
d
33

---

be constants. If m is large enough so that
32N C Rt
2N1
m 2
log
(280)
X
then with the probability of at least 1 over the initialization, we have
G (w (0) (0)) G | Ao
(281)
Proof. We use Hoeffding's 's inequality to measure the error between Go (w, v) and G. as Gram
matrix G is the expectation of the sample mean. In the Hoeffding's inequalities 220).
(228).
223
(241
245
(25) and 254 we have
d + 16 1
1ET) d
4EE SC.. p4e
(282)
d
V
d + KE
(plS) + 1)= 2
"
2151 < C
(283)
ng
d
V
(d 1)2
(d+1kl)
2 " p 211122 - C
(284)
no
d
v
p2 + +1) 209
(285)
n
V/Ve+1 2216 S C2,
(286)
n
1/VV
(d 1)²
m p El+2 < C2,
(287)
12
(d (d+1)
maxtt D* < CC..
(288)
n
Consequently, all inequalities induce that each component of G. - G, greater than E, with
a probability of at most 2 exp
me² 245
Since there exists at most Nj nonzero elements in
G. G by 145)
G (w, v)
G SE
(289)
holds componentwise, with the probability of at least
( 11- eex (-
ms²
m 2CP RAp
22
21 111-NN exp
exp
(290)
Then, we have
Gr (w,, (w, v) - G -
| Go (w.
G
S NN.
(291)
Set and to satisfy I/N, do and 2.V exp
ms² 20 R
< 6. In other words, if
2C RCP
RAP
2.N
m N
log
(292)
2²
8
32N, Of RAP
2NN
log
(293)
then Go (w, v) G < Ao with probability of at least 1-ō.
By the above proposition, the initial Gram matrix G, (w (0), v (0)) is likely to be strictly positive definite, and its smallest eigenvalue remains greater than de 2. at the beginning of the flow. The following lemma implies that at the early stage of the flow, for which w and v are not far from the initial values, the positive definiteness preserved, and its smallest eigenvalue remains distinct from zero.
34

---

Lemma C.5. Suppose - - we (0)ll2 S Rw < R. for
Aa Rw
Viaxx v 1 1 a max
(
(dttk)
(d+ lsl
p2101+1 (21)
(294)
8v2
d d
Then
G (i,, 0) Go (wr (0), v
(0))
- $4m $²^0-
(295)
Proof. 1. Case of Q0.o: By the mean value theorem with (151). induce that
(Qo.0 (w, 8) Qu, (w (0), (0)))
(296)
$ 20,0 (io, ") - 20 (w (0): 1 0);..e 11.63
+
Co.0 (w, i) Ro.0 (2o, v (0)) i1.12
(297)
θ dw
Qo.o (w", v (0)) (w - w (0))
+
Ro.0 (w (0), v²) in (0 v (0))
(298)
(Qo.o) (w0" (0))
-1(2),,, - -(wz), (0)1l2)
(299)
a (we)r
8=U r=
1 1000
m Keel |Jaeccal (1((02)rl12 12 -I((we), - (wz), (0)1l2)
maxx
p 2111+1
(300)
(wi) lies on line between w and w (0) and for some o² lies on line connecting ú and v (0). Since I[(w))))l2 S [(w)), (we), (0)llz + Il(we), (0)ll2 S R R < 2R for each & {0} U [L] and r e [m].
for some w [(wr]]]
(w)) T
(w
(Qo.0 (w, B) Qo,0 (wo (0), v (0))) 51.82
< mmax 211 mno
Tee Jasess 1 ((27) ((we)) (we)) (0)1.)
[d+ 1El 2 Σ
(301)
mmax p 2|4
Leel | (2R)22-1 R
No
t=
< max 2|
no d )
(d+ kt
(d+16l) (2 R.
Therefore, we can attain that
Ro
Ro
(oo,0 (ws, 8) Q0.0 (w (w) (0), (0))) 2,
12=1
(302)
d
k 2
+ |E1)
2p 222111)) (2 2(2p-)))
R2
d
Case of Qo,:: Similar to Q0,0+ the mean value theorem with 159 and 163 gives
QQ. (w, 5) Qo (w (0), u (0))) in (((a,)).5))
(303)
Viax
Omax
(d+15) 15l d
+1
|| (w0 2p- 11(00111)))) - (wi-1), (0)
(304)
mn.
+ (wv), 22--1 [(x0)), - (we), (w01)
(305)
/Vmax Umax mn,
(d (d+1s)
p 21
Σ 2 (2n)² 20-1 Rw
(306)
r=1
35

---

V/mxx
(a= p ((R)) Rw
(d+1s)
mmax
(307)
n
Consequently, have
Bo
Σ
EES Σ
(Q,
Qu,e (w (0). v (0))) 2
(308)
(1.((a.)).aa)
$1,2==
aElse-1 AETase
2
d (d+19) 1El)
max
22(2111+1)
Σ Σ
4(2H) 2(20-11 R2
(309)
1
2
d
i 1 8=1 aclse
1 BETTE
( (d+1El)
2
L
<V max max
p 2221e1+11
He | 4 (2 R
(310)
(S) d+k) d 1
SV² max max
(2R) 2(2p-1) (2R) R
(311)
3. Case of Q0,4.1: The mean value theorem with 171 induces
((Qo,L-1) ē- - (Qo,L+1) (w (0) w (0)))..
VVmax
d 15) d
amax (d+
1191+2 Σ (1( 2p-1 [( - (wo,, (0)1))))))
my/nomb
(312)
VVmax 2
d (s) Von max (d 11
pllt+ (2 Rw:
d
and hence 7i
no
((Q0,4+1) (w) (Qo. (wo (0) (0), (0))
(313)
2 Symaa (d+ + 1)² () (5+ 206)+ (2R) R
4. Case of QG,e, From the mean value theorem with (80 and 184 attain
((Qr,)) (w, δ) (Qr.c) (w (0), (0))) ((a) (1)
(314)
m
Vmax p² El+1 mn.
Σ
1 (01= (( 12 2--1 (( (wi-1),
(315)
+1 +al [((0z)),) 2 (( (we), (0)1l2}
(316)
Vmax p²
El+1
(|a=a0)) (2R) 2p-1 + 1(01t8 (2R))
20 } R
(317)
and further induces
L Σ Σ ΣΣ Σ
E
((Qc,) (w, 6) (Qe,e) (w (0) v (0)))²
((an (3)1))))
11 t=1 al P BeEaa
(318)
Su 2(2
3
3
Σ
Σ
Σ
{ (2
(319)
E=1 an f ETETa a2Elez-1 BeETaer
+1 +14an + (2R) (212)2(20111) 1)32 2 R²
(320)
S 2
Σ Σ
Σ
(1Isge - (2R) -2 + (11sel - 1) 12R))
(321)
36

---

+4(2R) (00))2(221) 11) R
(322)
SV 2 +1)
Σ Σ
Σ
4 و|ام (2R) 2(22--1 R
(2R)
(323)
1=1 Cl BETTae
<4v 2(1
2(211++1)
T Tasee 1 (2R)28- R
(324)
<4v max?
(4d 2 (2(()+1
(d+ k.
d (d+16l)
d ) (2R)28-1 R
(325)
d
192
follows that
(QQ,,4)) (io, 8) - (Qr,41) (wo (0), (0))) ((0.,)))4))))(00003)3))))
(326)
m
1 fag=an +D+
Mmax p |E++1
Σ
[1( 22-11 (((o))) (wx), (0)1l2)
(327)
mno
Vmaxx $ (ag=0r+81))
p (61l1 (2 2p-1 R.w.
(328)
no
It deduces that
The
L-1
Σ ΣΣX Σ
Σ
((Qutt)) (ūr, (Qe,41)) (0), (0))) 2
((a) (a..))
(1,12=1
t=1 an BIElaSE azElse
BBELEEE11
(329)
4-1 E
SV2 Sumax 21+2
Σ BI agEle, P2Elace4s
Σ
Σ
Σ
Σ
1 (02=ar+ (2R) (2R)212-11 R²
2(2p-1)
(330)
t= Ela
<1z <Una 21142
Σ
Σ
Σ
| 1/ (2R)2(22-1 R
(331)
t= alE-ss-1 BIEJaCE
h-1
-1 26122
Vec.. |Teeel | (2R))(20-11 1) R2
(332)
t=1
dd (d+111) (
d (d+ k) d
(
2 (2R) R2
Su² Sunax 211122
(333)
In accordance with the definition of the loss μæ, is clear that Q is identical the zero matrix if the difference between & and Eg is greater than 1. Consequently, together with (325 and (333) we have
Σ
Σ
Σ
Σ
no
((Qu,)) (0), (0))) 2 ((on ((o.)1))))))((oo000.)..)))
(334)
f122 1 an 1
P lase h1,2=1
no
L
Σ
ΣΣΣ
Σ
((Qx,) (w0, 0) - (Qe.e) (w (0), (0)))² 2 ((o) (()))))))))
in 1sl E 1 Brlsellace
(335)
1-1
+2 33Σ3 Σ
((Qeet)) () - (Qe..4)) (w (0), (0))) 2
((.) (()).))))))((())))))))))
hi F=l a BlEla arEler
BrELssent
(336)
(d+k) k)
(d+1kl)² 2 (2 R2
<4F² D ((2|g1+1)
(337)
d
37

---

+ 20 maxt
2111+2 21611
kdtk) k)
d ( (91 (
d (2R) 1) R2g
(338)
d
(
d + (4)
<6
22(2 (2R)
20-1 R
(339)
max
d
d
5. Case of Qe,++1: From 201 we have
((Q1,4+1) (wo, 8) (Q1,z+1) (w (0), (0))) (1a.9).0.
(340)
Vmmax
amax (d + 1) p15
(1( 20-1 [(wo), - -(wo), (0)1l2)
(341)
r=1
Vmax
Omax (d + 1)p 100p(111 /1+2 (2 20-1 R
(342)
By the definition of the loss pe, we have Qe,l+1 is identical to the zero matrix for every & > Therefore, we have
L
Σ ((QE,).)) (00, 8) - (Q1, (w (0) e (0)) 2
(343)
(1.0).0))
1=
8=1l BETARL
1h my
Σ ((Q114+) (wo 0) (Q1,L+1) (w (0), v (0))) 2
(344)
(0.9)))))
j=1 BETAC
< TAG Viaxx"aa (d+ + 1)2 (201)+22 (2R)
(2R) 2(2p-1)
R²
(345)
(d+ - 1
SV² max az amax
(d 1)² 2111+2 (2R) 1) R
(346)
[
Su 2 max
(d 1)²
dd+ (s)
2011+2) (2R) 1) n²g
(347)
6. Case of Q+41,411
From
205
we have
((Q (w. -(Q4+1,(+)) (w (0), (0))) 21
(348)
V
max (d+1) yd Σ (1(00)) 122 2p-1 [( - (wo), (011.)
(349)
mab
r=1
Vmmax max (d + 1)2 pd (2R)* 2p-1 Rw.
(350)
nb
further follows that
ny
n9
Σ
((QLI),41) (w, (Q1+1,1+) (w (0), v (0)))²
(351)
1=1
Su Max (d+11 pº (2R) (2 2(2p-1) 1) n²
Therefore, combining
302
311
313)
339
347
and 351
concludes that
G. (w. σ)
G. (w (0) v (0))
(352)
Go (w, 0)
G. (w (0) v (0))
(353)
dd* k
d + (4l p 2021))+1 (2R) 2(2p-1) R²
<amx
(354)
d
d
(d k)
(
d d 151) (021)++1) (222)222011 1) R2
+ 2
(355)
d
d
38

---

+ 1mmxx" (d (d+ (d+14))
(d= 1 (2111+2) (2R)2(22-1 1) R2
(356)
k
d ) ( ( (2R) 20-1 R
+ 6v max 22211111)
(2R)20-1
(357)
d
+ 22 "mxx max (d + 1) d [5]
p 211)))2) (2R)2(20-)) 1) R2y
(358)
+ maxx maax (d 1)*² pº (2R) 2 2(20-1)) 1) R
(359)
d k) /d (d+111) p 222 (21) 1) 12
<8v (1 + mmaxx
(360)
d
A
(361)
16
The preceding lemma indicates that the loss will decrease rapidly in cases where w is not significantly distant from w (0). Indeed, the subsequent lemma and proposition demonstrate that w (t) remains within the designated region for any given value of t > 0, provided sufficiently large m.
Proposition C.6. Set constants C2 and C3 as
C2 = Vmmax (11 + 0 )
d k)
d K5) d
(plel + +1).
(362)
1 ny
n6
C3
(x:)
9 (i²)²
(363)
1i i=1
1b f=1
+ kb
d +
[E
N₂ =
no + 1hb-
(364)
d
d
C2
For any C' > 0 and 8 < 2N2 exp
if m is large enough so that
204 Rev Rey
m 2
2C²
C2
4
(at
+ k)
d d
151)
RAP log log
(2N2
+
C2RRP)
(365)
d
8
then, with the probability of at least
over the initialization, we have
s (w (0), v (0))
11 (w (0) v (0))
1
d R
(366)
m
C
Mr (w (0) v (0)) hww (0), 1 (0)
Proof. Since
(d+h) dd 1sl
d d 12161 <C
(367)
d (d+k) d
d + El (pls) +1)² S CC2 d
Vinx
(368)
Uinaxe max (d (d+11) 1)) p² C2
(369)
we have
m
22
P
Σ
Sir (w (0))
E
< 2 2eep
(370)
Vno
2C RiP
2²
P
Σ (He) (w ()))
E Vno
< 2exp
(371)
2C Rep
39

---

m
E²
Σ
hyrr (w (0) v())
S 2 exxp
(372)
202 1t3p
Tbb
Hence, from
L Ro | . |Tecel . no + ns S
+ k:
(d (1+1
no + ny S N2,
(373)
d
d
with the probability of at least
(1-2 2exp
203 20310P)) 10e9) ))
8²
N₂
82
21- 2N exp
(374)
203 122 1020
all the conditions
m
S/r (w (0) 1 (0))
1 E
(375)
noo
m
(Pe)nta.00 (w (0), 1 (0))
<
E
(376)
nno
her (w (0) v (0))
E
(377)
Thh
are satisfied. Then, the square of the initial loss is bounded as
|s (w (0), a (0))12 + ||h (w (0), a (0))12 +
Σ
M (w (0), (0)11
(378)
t=1
R2g
Σ
st (w (0) 1 (0))² 2
hy (w (0) (0))
(379)
Σ
(me) (w (0),
(380)
1=1 f= ac vE
RETASE
ng
ng
( + 1f (2:))) +
Σ
(s+ 1g (ū, (2))))++
+
|ec-1 1- MAaecl2
(381)
11
Y&b
j=1
t=1
1
d (d+R k)
d |s]
(E+ 1f (x:))) +
(s+ 1g ( )1
+
E²
(382)
n1 1=1
d
d
< 4
d k)
d ))
(
(a+16l) 8² +2
(d
1 no f (xx)²
2
> 9 (3g)²
ns
(383)
d
no
Thb
(d (d+H) kb
(d+ (5) 22 2C3.
4
(384)
d
For 8 > 0 and C > 0, if e and m satisfy the following inequalities,
E²
8 2 2N₂ exp
(385)
2C2
E > C2.
(386)
C R2v
(4+
(a
kb) k.
dd+ (d+1s)
)) e² #+2Cs)
m. 2
(387)
X
d
d
then for enough small 8 such that
C2 222 1220
< 2N2 exp
(388)
40

---

we have that with the probability of at least - 6,
2 (0)) + || (w ( (0)) ||s (w (0)
Me (wr (0), v (0))
(389)
m
t=1
d + ) d (1=11) 22 203)
4
(390)
A C
R2
(391)
In other words, if
2C
d + h:
d )
d (d+1s))
2NN
C2
4 +
R log
+ C2R2v)
(392)
with the probability of at least
we have
(w (0) (0))
P1 (w (0)
Ao R.P.
(393)
m
20
Mz (w (0), (0) (w (0) (0)
Proposition C.7. Suppose the conditions
281
holds and Re are given as in Lemmna
C.S
For
d+k
15l
C4 C==611 a max Umax
p
11
|sl
2r
(394)
d
d
R
if m is large enough so that
(w (0) v (0))
(20 (0),
1
do R-P, P.
(395)
C4
My (w (0) 0 (0)
h (w (0), (0)
then
[[(=;), (t) (wz)), (0)ll Rw [1(ve), (t) (ve), (0)] < R
(396)
(397)
for all 0, [m]] and = 0,...,.. L.
Proof. We begin the proof by defining
T sup {t 2 0: [[(ww), (t) - (we), (0)ll, R. and [((e), (t) - (wv), (0) < Rw, VC 0,... L}.
(398)
Since (wz), and (ve) are continuous, the above set in supremum is nonempty. For t (0,T),
s (w (t) (t))
M1 (w (t), (t)
d
(399)
dt
M1 (w (t), (+)
h (w (t),
s (t) P (w (t).
s(w v 4 (w (t), (t)
G GG.)
(400)
#y (w (t), h (t),
Hy (w (t))
(w (t),v)))
41

---

8 sw (1) (1) M1 (w (
2-2 1 A
(401)
Mr. (w (t), h (
where the last inequality comes from Lemma implies that
C.5 and the positive semi-definiteness of G This
(w (t) v (t))
(w (0) v0 (0) (w (0),
M1 (w (t)
2o,
S eexp
(402)
Mg (w (t),((tt))
pL (w (0), (0)
h (w ut),
h (w (0), (0)
For / E (0, and é E (0,... LJ. from (w=:), (0)ll2 - R and [[),)), (0)ll2 S 1/cl and (388. we
have
((we), (f)122 [1(w), (t) (we), (0)ll2 + (1( (0)ll2
(403)
Rv + R
(404)
$ 2R,
(405)
(((ve), (t)ll2 < (1(ve), (t) - (ve), (0)ll2 + (1(x)), (0)ll
(406)
< Rw + Meel
(407)
R.
(408)
From 44
57
52
and 48
can attain
0s; (w (t) (t))
Omax V [Tg, JACC+1 124 R"
(409)
0 (we),
mno
Th, (w (t) v (t))
Vmax
amax (d + 2220 RP,
(410)
(wvo),
mn,
1 (a,)) (w, v)
Vmax mn,
p2" Rp
(411)
① ((Ws),
1 (He) (w,v) v)
Vmax
pl +122-112 R²,
(412)
1 (w
mno
for all t E (O, Therefore, can induce that
d
(we)) (t)
(413)
dt
Os; (w (t) v (t))
Th
0hg (w (t) v (t))
(w (1), (1))
hi (w (1),v())
(414)
7 (we),
(wee)
i=1
j=1
0 (a..), (w ( d (werr
(He (HL)) (w (t) u (t))
(415)
of oETep_1
BE DETSE
Ro
Os: (w (t) v (t))
18: (w (1) (!))
(416)
① (ure),
2
Oh; (w (t) (t))
+ 1(6=0)
[/y (w (z),
(417)
0 (wr),
42

---

① (w (4), 2 (t)) 2 (wee)r
ΣΣ 1 +
(1)) (w) (1),)())))
i=1 a RETasr
(418)
(1&+1) (orR), (w (1), (t))
ΣE
(mett) (a.2), (w (t) v (t))
+1(()))
δ ((we)r
i=1 aelge BElsee41
(419)
The first term and second term bounded by
Us; (w (t), v (t))
|s:; (20 ())
(420)
a (we)r
2
Vmmxx mnno
amax V Ugel JDsc+1 p111+120-119 RP, |s (w (t), (t))
(421)
1 no n2o
Vmax m
amax
|Toel Jasenil p
1211 Rp.
8: (w (t) 0 (f))
(422)
Vmax
amax
V MEell
JAseral /p 12 Re
||8 (w (t), (1))l22
(423)
and
no
Th (w (t) 2 (t))
1b, (w v (t))
(424)
á (we)r
Vmax
Omax m
( 1)
R2.
1
1b (w (1))
(425)
j=1
Vmax
mmax (d ()+1) R | (w (
v (t))
(426)
m
The third term is bounded by
1 (Wv) (w (t),
[He) (w (t), (t))
(427)
① (wE),
1=1 No
BETTEL
Vinx
p2 RD
(He)a (w (
(428)
mno
1=1 aE 2
BETSEE
Vmax m
p22 1 Rp
/ |TEeel 11pel22
(429)
and the fourth term is
2 (Met1) (on)), (1) I O(we),
ΣΣ Σ 1=1 aETE, BET
(pet)) (w (t), 10 (t))
(430)
SV mno
Vmmax 111l+120 Rp
Σ
(Mett) (w (t), (t))
(431)
nETe, BElace+s
Vmmxx
m p 12 PpP v /eell -Aseial ||Metallz.
(432)
Therefore, for L E TD, and é =
d dt
(une), (1)
(433)
2
43

---

Vimx
umaxy/ /
(a+ k)
1111-120- | (w (t) v (f))l2
(434)
d
d
+ 1 (1=0)
Vinax
Omax (d + 1) p2200-1 R2 1 (w (t), (t))ll
(435)
Vmi
Vmmxx
d ki d
+ [E]
+ 1 (+*00 V
p2 R | (w (t), e (t))ll2
(436)
m
d
+ 1 [L#L)
Viax
d d
d + [E]
pl 111l-120111 Rp |1 (w (t) v (1)
(437)
(w (t) (t)) M1 (w (1) (()
Vinax
(d+h) h (dd d
(d+ p pl R2
<3
1 + a V
(438)
m
Mr (w (1), (t))
h (t), (t))
2
u (0) 10 (0) (0) (0))
14 (w
Vmax
dd+ kb
dd+ (d+1el) plett12 RP exp
(-xo, (-221 22)
<3 1 + o max
m V
d
d
b2 w . (0))
h (w (0) (0)
(439)
On the other hand,
V1-d ds
d =
%/ ds
(we), (s) ds
Σ
(we) (8) ds
(440)
2
9=1
(we)
(8) ds
(ure)
(s) ds
(441)
ds
ds
q=1
d
0 ds (ure), (8)
ds
(w))
(8)
ds
(442)
0
ds
2
1/"
d
T
d
(ws), (s) ds
(wi)
(8)
ds,
(443)
ds
0
ds
2
and hence
[(we), (T) (wx) (0)llz
(444)
d (wz), (8) ds
(445)
ds
d (we), (5) ds
(446)
ds
(w v0),v())
pn (w (0))
<3 1 azaxx
Vmmx
(d+h) k.
d (d+1e))
pl 1114 Rd {" exp
No v ds 2
m V
d
1y (w (0) 1 (0))
h (w (0) (0)) (447)
44

---

(w (0))
#1 (w (0) (0))
S v 1 + amax
6
Vmax
d dd+b) kb
d (d
lEl
plll+120-1111 RD
(448)
A0
m
d
d
Mz (w ( (0)) w (0) (0))
If m is large enough to satisfy (395 we have
6y + " v Mmax
(d++)) (d+16 ) p
Ao 1
(w (T) (we), (0)ll2 -
Ru
Rw-
Ag
C1
2
(449)
Similarly, we obtain
d
(ve), (t)
(450)
dt
2
no
Ds: (w (t), v (t)) (ve),
The
hy (w (1)x(()) Thy (w (t) v (t))
Σ
8i (w (1) (1))
(451)
(
j=1
0 (wee) (He)) (),
(w (1) (1))
(Hr) (w (t), (t)
(
af 1 BET
(452)
3s: (w (1) v (1)
88 (w (t), 1 (t))
(453)
① (ve))
Ohg (w (t), (t))
+ 1 ((=0)
>
|by (w (1), (0))((1)) (1))
(454)
(ve),
/=1
a (He))a (w (t), (t))
+ 1 [ex0))
Σ i=l aEter- BElaE
Σ
(Ho) (w (w), (t),((t)))
((v)),
2
(455)
no ΣES
(pz+1) (a (t), (t))
Σ
(Meti) (w (f) v (t))
+ (((LL)
0 (vo)r
f=1 aEls BElase11
(456)
The first term and second term are bounded by
0s; (w (t) v (t))
sss (w (t), v (t))
(457)
a (ve)r
Vmax
amaxx]
d+ k d
d [5l
ple (2R)²
1sa (w0 (t) v (t))
(458)
m
d
/no
(d+ k) d
(d (sl)
Vmax m
m
1lld (2R)" ||
(459)
and
Oh, (w (t) v (t))
h (w (t), v (t))
(460)
i (ve),
V/max m
(d + 1) OmaxP (2R)
I (w (t), (())
(461)
7hh
45

---

Pimxx
(d 1) AmaxP (2R) |/ll2-
(462)
m
The third term is bounded by
① (1e)
(w (0) θ ((v)),
Σ
Σ
Σ
(He)) (w (t),
(463)
aEdee
BELaSE
S mn,
Pmax
d k.
d (sl)
no (2R)²
(we) (w (1), (1))
(464)
aEle,-1 DElacr
Vmax
d k
d + (5)
(2R)" /Ie.lll |1 ||eell2
(465)
m
Vmax
d + k)
d + (sl
(2R)* 1|xel12
(466)
and the fourth term is
Th
① (Met1) (a,0), (w (t), (t))
E Σ Σ
(peta) (w (t), v (!))
(467)
0 (1v)),
1= aflee BElaci+1
Vmmux mno
pe
(
d dd*k)
d d [s]
(2R)* Σ Σ
(pett) (a. (w (t), (t))
(t))
(468)
aETEz BE TET
Vmax (d+k) (d (E)) 1lll (2R)² (1t))11|e4aal2-
(469)
m d d
Therefore, for / T) and = 10,........ [0,...,L}
d (v)), (t)
(470)
dt
Vinx m
Omnx
+ k
d lsl
plsl (2R)² ||*||2
(471)
d
d
+ 1 (c=0)
V m
Umax
(d + 1) amaxp (2R)² ||All2
(472)
Vunax
d d + [5l
+ (1y0)
(2R)" ||ee||2
(473)
d
(e42)
Vinx
d + k:
(d + [gl
pll (2R)" |c+11l2
(474)
m
d
d
8 (w (t) 2 (t))
pi (w 20 (t))
<3
Vmax
1 2 amax
kd+k)
(d [El d
plkl (2R)²
(475)
m
Me (w (t), (t))
h hww (1),
8 uw (0) 10 (0))
M1 (w (0) (0)
53/ m
Vmax
V + a2axx
(d+ ko
( (d+ [sl
d plll (2R)* exp
A0 (-20)
d
2
P1 (w (0), (0))
h h(w (0) (0))
(476)
As consequence, we attain [((x), (T) (ve), (0)llz
(477)
46

---

( d ds
(ve), (s)
ds
(478)
8 w (0) v (0))
11 (0) (0))
/31
V/nxx
1 + max
dd+k) ( (d+ (d+15) [5]
pl5) (2R)*
/
T
exp
(-
ds
2
P1 (w (0) (0))
h (w (0) v (0)
(479)
(w (0), (0))
11 (w (0),
S V 1
6
0max
Umax
) m d
d k k)
d (d+16))
d p (2R)²
(480)
Ao
0 Ng (ur (0), (0))
h (w (0), (0))
Since m is large enough to satisfy 335 we have
[[(=)), (T) (ve), (0)Il, - 6v1 + a amaax v Mmnax
(d+ (dttel) 1l
(
Ao
- Rh Ru (481)
1
Ao
C4
C.1 Proof of Theorem
Proof. For R E R, E [m]] and E [d], each (we)rip satisfies
(-2 1t2
P (we)no > R
exp
(482)
If we set 81 (L + 1) md exp (-3 R?",. then 81 satisfies
I 1-exp (-)
(L+1)md
21 +L-(++) mddxp (-1 = -61 (483)
Hence, with probability of at least 1 - 81, we have (wwerro r E [m]] and E [d].
S R for all l = {0, 1, L 1}.
Provided (we)r.g S for all {0, 1, 1),
32N, PPPP
2N
m N
log
(484)
1
82
where
C
d k)
dd (S)
2
2 U
+
1
/1 +
a
(D2tE + +1),
(485)
191
no
17b
d + k
dd + [
(d
+ |g]
Ni 7
n
2
n, + n
(486)
d
d
d
then by Proposition C.4 with probability at least - 82, we have
G. (w (0), v (0)) - G ~0 || < Ao
(487)
4
On the other hand, with the condition
(we)
< R for all l =
10.1,...L+1,,
11. 1).
[m], and
E [d], Proposition C.6 shows that if
m 2
2C?
C2
+
d ++k k)
(d+1k)
1))
R log
2N2
+ C2AAp)
(488)
2
d
d
47

---

C 2C R2p
83 exp
(489)
Rw R
(490)
where
Ao 8v2
d+ (d [d+1el)
d (+))))) p21l1+2 (2R)
141)
R
Vmmax V 1 +
a max
20-1
(491)
d
d d+k)
(d (d+11)) 151) (p151+1)
C2 Vimax
(1 + a
(492)
d
d (āz)²²
C3 =
(zx)) 2+ +
(493)
1bb j=1
C4 VVmax V 1 a
d + k
d + 151
pl
(494)
d
d
R
N₂ d d+
d + |5
no Th::
(495)
then, with the probability of at least
1 d3. we have
s (w (0) v (0))
11 (w (0)
1
d R.P.
(496)
C4
[& (wo (0), (0))
h (w (0), 12 (0))
Consequently, Proposition C.7 implies
((w=)), (t) (wzz), (0)1 Rw [1(ve), (t) (ve), (0)ll < Rw
(497)
(498)
for all allt 2 0. Then, we have
s (w (t) v (t))
(w (0) (0))
#1 ww (t))
#1 (w (0), (0))
S exp -
(499)
My (w ( h (w (1),
My (w (0) v (0)) (w (0) v (0))
2
from Lemam C.5
Set 6, = 82 83 = 5 and consider all inequality conditions. First, for 8, 48.
6 (L + 1) md
R
log
(500)
5
satisfies the condition 483) Then, 484
is satisfied if
32N, C?
(L + 1) md
2p
6NN,
m
X
log
6
log
(501)
Similarly, if m is large enough so that
20207
d + k.
d (El)
))
6 (L + 1) md
2p
2N
4 +
log
log
(502)
X3
d
d
8
8
20
6 (L + md)
log
(503)
XB
8
then 488 holds.
48

---

Therefore, there exsits C₃ = Cs (Amax: Vmaxx No. Ro, Ao) such that if
m + kb
6 d |sl) 8 p 2 1og md 4p
(504)
then with the probability of at least (1 81) (1 - 62) (1 0a) 2 1 (61 + 82 + 03)
6, we
have
8 (w (t) v (t))
8 (w (0) 2 (0)
14 (w (t) 0 (t))
1 (w (0),
(- 2o (-2o, )
$ exp
(505)
Mg (w (t)) h (w (t)
μy (w (0).0 (0))
h (w (0) 1 (0)
D Experimental Details
D.I Problem Formulations
Poisson equation
Uzz + Uyy = f1 n (1x 0) a (1r, 1T) (0, y) (Tr, v) 2r y [0, m)
2, (0,
(506)
where f1 =0, + R defined so that the exact solution is given by
a (x, v) sin" (x) (T y)²
(507)
Bi-harmonic equation
Uzzes + 2uaayy + Myyyy = f2
I,y/ E (0, v, E [0, T
u (z. 0) (z, T) (0. v) (T, v) = 00 0 8n u (I, 0)
(508)
9 u (I, R) ön (0,
i u
(-,
I,y E (0,
ön
where the flux function f2 : (0, + IR is set to ensure that it has solution that is identical to that of Poisson equation (507)
Heat equation
14 Uza (t, -1) u (t,
E (-1,1), LE (0.1) te [0.1] xE (-1,1),
(509)
u (0, x) sin (TZZ)
Elastic beam equation
+ Unazz = 0 a (t, = (t, σ) = Usz (t. U2x (t, 0
x E (0,T),
T), t (0,1)
te [0, 1]
(510)
u (0,z) = 2 sin (x)
x E (0,
The exact solution is given by u (f, = 2c 2e-1sin sin (x)).
Confection-diffusionn equation
Ut + Us furz =0 0 u (0, x) = sin (x)
(t,z) E 9 - (0, (0,z),
a E [0,
(511)
u (t, = -e- : sin (t)
te [0, 1],
utt, TT) - sin (IT
i E [0, 1),
whose exact solution is u (t) - sin (x t).
49

---

Table 2: Experimental settings for each PDE
PDE
width m
power p
optimizer(I))
no
R6.
Harmonic
10a
109
5
GD(10-8)
400
400
Bi-harmonic
10
10
~
GD(10-*)
400
400
Heat
Adam(10
300
Hea((PNNNs)
109
GD(10-1
300
300
Seaa(VS-PINNN
2
GD(10
300
300
Beam (PINNs)
5
Adam{ 10
10,000
300
Beam (VS- VS-PINSS < 2)
10a
3
Adam(10
10,000
300
Beam (VS-PINNS 1)
10a
2
Adam(10 (10
10,000
300
Convection-diffusii (PINNs)
10a
3
Adam(10 / GD(10
300
300
Confection-diifuuionn (VS-PINNs)
10a
2
Adam(10-
-)) GD(10-1)
300
300
Bi-harmonic (PINNs)
10a
5
Adam(10- -3)
10,000
400
Bi-harmonic (VS-PINNs
10a
Adam(10 33
10,000
400
Bi-harmonic (VS-PINN) 1)
10a
Adam(10 3)
10,000
40000 epochs
Total 400000 epochs
PINN-O-001
1
PINNsl10,00
VS-PINNad--0.)
VS-PINNa(--O.1)
0.1
0.1
Train loss
0.01
Train
0.01
0.001
0.001
0.0001
0.0001
1o405
100
300
350
400
500
1090
1500
2000
2500
3000
3500
4000
Epoch (x100)
Epooh (x100)
Figure 3: Loss of convection-diffusion equation trained by GD.
D.2 Parameter settings for experiments
Convection-diftusion equation We conducted experiments on a convection-diffusion equation: We train 100,000 epochs of PINNs with p and VS-PINNs with p using the same settings as represented for the heat equation (509). Figures Baddd showw that VS-PINNs reach lower loss and achieve more stable convergence for both GD and Adam.
Bi-harmonic equation
Consider the bi-harmonic equation
A A (x. f f (x. (I, 0) u (I, u (0.y) ( = (T.
2, [0,
2, [0,
(512)
u (z. 0) am (z. x) am (0.
2, E [0,
where f (x, y) are defined so that the exact solution is given by u (x. y) sin? (x:) y² (N y) 10,000,
400, and the training collocation points are fixed once they randomly selected. We experiment VS-PINNS with two cases: (i) do w. ΦI (umay Uyyy) with |g| 2 and p 3 and (ii) the finest splitting of Φn ~ u, Φi s Vu, Φ2 RJ (uams Upy 63 = (Uzar, Uzzy. Myyy) with [ and p Regularization parameters are 11 122 1/3 for derivative matching loss f1 and 10 for boundary loss h. Figure depicts the training loss of PINN and two VS-PINNS.
We set m = 1,000,
E
Additional Experimental Results
Computational efficiency of VS-PINNs Table 3 measures the GPU memory, running time, and the number of model parameters corresponding to experiments on elastic beam and bi-harmonic equations. Because VS-PINNs need as many networks auxiliary variables, finer VS-PINN requires
50

---

Initial 10000 epochs
Total 100000 epochs
PINNs VS-NINNs
VS-PINNS
0.1
0.1
0.01
0.01
Train loss
0.001
Train
1c-05
0.0001
1c406
1e-05
1c:07
100
200
400
600
880
1000
Epoch (x100)
Epoch (x100)
Figure 4: Loss of solving convection-diffusio equation trained by Adam.
Initial 40,000 epochs
Total 400,000 epochs
PINNS
PINNS
100.0
VS-PINNS 1ET
100.0
VS-PINNS [E] -
VS-PINNs 15
VS-PINNS [E]
10.0
10.0
Train loss
1.0
Train
1.0
100
300
350
400
500
1000
1500
2000
2500
3000
3500
4080
(0pochs(x100)
Poccsssx 100)
Figure 5: Loss comparison between PINNs with two VS-PINNs with |g| = 1 and 2.
more parameters to be trained. However, reducing the differentiation order in the Loss significantly reduces memory consumption, despite the additional loss terms.
Error with true solution As the ultimate objective of PINNs is to approximate the solution to the PDE, we also quantified the discrepancy between the exact solution of the predicted solutions of the three models we tested. tested five times for each PDE and splitting setup from different random seeds, and the averages of the Mean Squared Error (MSE) with the exact solution are reported in Table 4 We can observe that the results do not exhibit the same tendency as depicted in Figure 2 Since there is discrepancy between optimization ability and the generalization error, it is not within the scope of this paper, and we believe that this is an important area for future research.
GD results for fourth-order PDEs order to address fourth-order PDEs (elastic beam and bi-harmonic equations), PINNs should employ p 5 of the power of the ReLUP activation. When optimized with GD, this configuration results in a loss value NaN, indicating that PINNs fail to train. Consequently, However, in contrast to PINNs, which exhibit difficulty training, variable splitting demonstrates superior performance in terms of loss reduction for GD, illustrated in Figure 6 This is attributed to the fact that variable splitting employs lower derivative order and lower power
present the results of our experiments with the Adam optimizer in Section 5
F Broader Impacts
We analyzed the behavior of PINNs when applied to general kth-order PDEs and provided theoretical understanding the reasons why PINNS encounter difficulties in optimizing when dealing with high-order or high-dimensional problems. This paper has societal impact as it bridges artificial intelligence and physics, and hence has wide range of potential applications. could contribute to the advancement in scientific understanding, and foster scientific progress and innovation, benefiting
51

---

Table 3: Computation costs with width=1000 for experiments on the paper.
PDE
Method
GPU memory
running time
parameters
PINN
801.052 Mb
0.053 s/epoch
4000
Bi-harmonic
VS- JS-PINN
1
2
481,094 Mb
0.049 s/epoch
9000
VS- VS-PINN
81,466 Mb
0.053 s/cpoch
20000
PINN
323.772 Mb
0.037 s/epoch
4000
Beam
US-PINN VS-PINN
2
240.689 Mb
0.038 s/epoch
9000
80.836 Mb
0.040 s/epoch
17000
Table 4: The average of Mean Square Error (MSE) with exact solution.
PDE
PINNs
VS-PINNs lel = VS-PINNs [&|
Elastic Beam 3.0598E-08
2.2868E-08 5.3055E-05
4.3826E-07 5.4418E-03
Bi-harmonic 3.9319E-05
researchers in physics. mathematics, and artificial intelligence fields. By providing variable splitting which can address the convergence pathologies of PINNs, the paper suggests potential applications in industrial and technological fields, such as engineering, medicine, materials science, and environ- mental modeling, for complex system modeling, prediction, and optimization. Furthermore, as many problems have arisen in sciences and engineering tied with complex PDE systems, we expect that our work has the potential to applicable in the enormous area such as climate forecasting, epidemies, molecular simulations, micro-mechanics, and modeling turbulent flows. Additionally, PDEs can be utilized in the development of military equipment. As with all numerical algorithms. however, is not work of developing technique go to warfare; rather, it tool for scientific inquiry. We encourage users of our model to focus on the positive impact of this work.
52

---

Initial 40,000 epochs
Total 400,000 epochs
Ir-0.01
80.0
Im:00001
0.00000
60.0
60.0
0-0.01
Train loss
Train
40.0
1- 0.0001
20.0
20.0
0.0
0.0
100
100
350
400
500
1000
1500
20000
25000
3000
3500
4000
(:pochs(x100)
Epocbs(x /00) l0))
Figure 6: Loss of VS-PINNs trained by gradient descent with variant learning rates.
53

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
54

---

Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
55

---

Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwticccguie//eeieeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
56

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ftttps://neurippe http///urrs.cccpuui////e http://...ccouocccttttcctttteeea
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
57

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
58

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
59


Reviews:
[{"review_id": "5apLOfT03P", "reviewer": "Reviewer_inw1", "summary": "This paper provides a theoretical understanding about the behavior of PINN when dealing with high-order or high-dimesional PDEs. Variable splitting is then proposed to decomposes the high-order PDE into a system of lower-order PDEs and facilitate the convergence of PINN.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper extends Gao et al.'s work to higher order PDEs and ReLU activation functions and demonstrates that the gradient flow of PINNs converges to the global minimum with a high probability when the width of the network is sufficiently large.\n2. Using the derived bound, this paper comprehensively analyzes the impact of PDE order and dimensionality on the behaviour of PINNs.\n3. The numerical results further demostrates the theoretical results and validates the positive effect of utilizing variable splitting.", "weaknesses": "1. The numerical results pertaining to the parameter  p  indicate a contradiction. Specifically, the convergence of the PINN is observed to be faster with  p = 4  compared to  p = 3 , which stands in contrast to the assertion that “the convergence of the loss is enhanced as  p  decreases.” Additionally, there are typographical errors in Figure 1(a), where the legends should display  p  rather than  k.\n2. The derived bound demonstrates the substantial impact of both the PDE order and the activation power on the network’s width, indicating that as  p  increases, the width  m  must also increase exponentially to ensure convergence. However, the numerical experiments conducted in the paper do not vary the network width  m  to empirically verify this theoretical relationship. Instead, they maintain a fixed width across different experiments, focusing primarily on the effects of varying the power  p  of the ReLU activation function and the impact of variable splitting. While these experiments do validate the theoretical findings related to  p  and variable splitting, they fall short of demonstrating how changes in network width influence convergence behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. The numerical results pertaining to the parameter ...\n\n**R**: The theoretical results presented in this paper indicate that networks with a lower power $p$ have a higher probability of convergence. A reduction in $p$ would facilitate optimization, which may be observed as an acceleration in the convergence process. It should be noted, however, that our findings pertain to an improvement in the probability of convergence rather than to the speed of convergence.\n\nWe acknowledge that the experiments presented in our paper could be misinterpreted as demonstrating faster convergence. To address this, we conducted additional experiments to provide a more robust validation of the theoretical results. Details of these experiments can be found in the common response above. The experimental results indicate that as p increases, a wider network is required for convergence, which is consistent with our theoretical findings. \nThese results will be included in the revised manuscript, and Section 5 of the submitted manuscript will be revised to clarify the existing experiment and avoid misinterpretation of the results as showing the effect of acceleration.\n\n**Q2**. The derived bound demonstrates the substantial impact of ...\n\n**R**: We are grateful for the reviewer's thoughtful review and for emphasizing the necessity for empirical validation regarding the relationship between network width $m$ required for convergence and power $p$ of activation.\nWe devised an experiment to validate the impact of power $p$ and PDE order $k$ on the width size $m$ required for convergence, which can be found in the common response above.\nPlease see the description and our discussion in the common response above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sthaXIoWKV", "reviewer": "Reviewer_M6m4", "summary": "This paper presents a theoretical analysis of the relation between PDE order and the convergence of PINNs. A tighter bound is obtained than the previous work. Inspired by the importance of reducing PDE order, the authors propose the VS-PINN, which employs the variate splitting strategy. Both theoretical analysis and empirical results are included to prove the effectiveness of the proposed method.", "questions": "-\tI suggest the authors provide a clear description of the proof for Theorem 3.2 and show how they obtain a tighter bound.\n\n-\tI think the equation in line 247 has a typo, which is $l\\in[0, L+1]$.\n\n-\tHow to choose $L$ in practice?", "limitations": "I appreciate that they have discussed the limitations. There are some more powerful and advanced backbones, such as PINNsFormer [1]. More experiments on them will further demonstrate the effectiveness of the proposed method although they are hard to analyze in a theoretical aspect.\n\n[1] PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks, ICLR 2024", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tOverall, I think this paper is theoretically solid and interesting.\n\n-\tThe idea of using variable splitting is reasonable.", "weaknesses": "1.\tPractical efficiencies are expected for comparison, such as GPU memory, running time and model parameters. Although VS-PINN does not require the calculation of high-order derivatives, the newly added regularization term in Eq. 13 will also bring extra computation costs.\n\n2.\tAll the conclusions in Section 3 can be directly obtained from the previous paper [22]. I do not think the tighter bound brings better theoretical insights in terms of understanding “How does PDE order affect the convergence of PINNs?”\n\n3.\tHow about model performance in typical PDEs, such as Convection or Reaction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Practical efficiencies are expected ...\n\n**R**: Table A1 in the attachment shows GPU memory, running time (the mean of the 50 epochs), and the number of model parameters that correspond to experiments presented in our paper.\nBecasue VS-PINNs need as many networks as auxiliary variables, finer VS-PINN requires more parameters to be trained. However, its reduction of the differentiation order in the loss significantly reduces memory consumption, despite the additional loss terms. Regarding running time, splitting does not increase the overall training cost.\n\n**Q2**. All the conclusions in Section 3 can be ...\n\n**R**: We want to clarify that the conclusions of Section 3 of our paper are not directly derived from the previous work [22], as the bound in [22] is not dominated by the PDE and the power of ReLU$^p$. \nOn the other hand, our paper attains tighter bounds than [22]. Moreover, the dominating term of the proposed bounds is determined by PDE order and activation power, allowing us to observe the impact on GF convergence.\nTo provide a more comprehensive understanding, we will delineate the leading terms of the bound of [22] and ours in the following paragraphs. We hope that the reviewer will carefully consider why our tighter bound is insightful.\n* (Leading term in [22])\nIn [22], Theorem 3.8 states that if the width $m$ is $\\tilde{\\Omega}\\left(\\delta^{-3}\\right)$ then GF converges to a global optimizer of PINNs for second-order linear PDEs. The leading term of the bound is $\\delta^{-3}$, which stems from the product of triple $\\delta^{-1}$. One $\\delta^{-1}$ came from the Markov inequality to bound initial loss at Section B.4. The remaining two $\\delta^{-1}$ are required for the bound of $R_w$ at Eq. (74) and $R_a$ at Eq. (76), which are also derived from the Markov inequality immediately following Eq. (73). Therefore, the leading term $\\delta^{-3}$ originated from the Markov inequality and is independent of the PDE order or the activation power.\n* (Leading term in ours)\nIn contrast to using the Markov inequality, we obtain the bounds in a deterministic way; refer to Proposition C.6 and Lemma C.5 for the initial loss and $R_w$, respectively. This yields a more precise bound for our Theorem 3.2, whose dominant term is contingent upon the power of the activation function and, thereby on the PDE order.\nConsequently, we obtain a considerably tighter bound that depends on the PDE order, whereas the previous bound from [22] was not affected by the order. From the relationship between the PDE order and the bound for convergence, we could derive a theoretical insight that reducing the PDE order enhances the convergence behavior of PINNs.\n\n**Q3**. How about performance in typical PDEs?\n\n**R**: In response to the reviewer's request, we conducted experiments on a convection-type equation. Since observing the effect of VS on first-order PDEs is unable, we convection-diffusion equation $u_t+u_x-u_{xx}/4=0$,\nwhose exact solution is $u\\left(t,x\\right) = e^{-\\frac{1}{4}t}\\sin\\left(x-t\\right)$.\nWe trained PINNs with $p=3$ and VS-PINNs with $p=2$, using same settings with the heat Eq. (18) in the manuscript.\nFigures A2 and A3 in the attachment show that VS-PINNs reach lower train loss and achieve more stable convergence for both GD and Adam.\n\n**Q4**. The equation in line 247 has a typo..\n\n**R**: The symbol actually denotes an integer set defined on line 531 of the Appendix, which did not clearly specify in the main manuscript. We apologize for any confusion this may have caused. We will make sure that the necessary definition is included in the revised version of the paper.\n\n**Q5**. How to choose $L$ in practice?\n\n**R**: We agree that discussing the optimal choice of $L$ is crucial for effectively utilizing VS-PINNs. \nHowever, identifying the optimal $L$ is challenging because training deep learning models involves a multitude of complex considerations.\n\nOur paper is focused specifically on one aspect of deep learning training: convergence of GF. \nAccording to our theory, splitting a given PDE into a system of first-order PDEs, such that $| \\xi | =1$, helps GF most likely to converge. This finest splitting results in a loss that includes only first-order derivatives, allowing the use of $p=2$, which in turn improves the convergence of GF the most.\n\nExperiments also show that breaking down a PDE into a system of lower-order PDEs tends to be most feasible for convergence. \nMoreover, Table A1 which was referenced in the previous response indicates that VS is an effective approach for optimizing GPU memory utilization.\nHowever, while finer splitting enhances convergence, it also increases the number of functions that need to be parameterized by networks. Therefore, in practice, the choice of L should be balanced based on the governing PDE and memory/computational constraints.\n\nFurthermore, it is critical to consider other factors, so we acknowledge that this is an area requiring further exploration for a robust analysis of the optimal $L$.\n\n**Q6**. There are some more powerful and advanced backbones, ...?\n\n**R**: We conducted experiments to assess the efficacy of our approach within the suggested PINNsFormer architectural framework. We implemented our VS strategy, conducting experiments based on the wave equation in Section 4 of the original paper of PINNsFormer [1]. All configurations specified in [1] were adhered to, including the use of the L-BFGS optimizer. To further examine the behavior of models, we also experimented with alternative optimizers, namely Adam and SGD, with a learning rate of 1e-4.\n\nThe results are depicted in Figure A5 of the attachment. As evidenced by the results, the VS does not yield a notable impact on the PINNsFormer framework for all three optimizers. It seems that the Transformer architecture is substantially different from the MLP structure examined in our paper, resulting in disparate outcomes and indicating that comparable results may not be attainable in this context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcaHExadbH", "reviewer": "Reviewer_AeaP", "summary": "**Summary:**\n\nThe paper discusses the convergence of the gradient flow of a PINN loss function for arbitrary-order linear PDEs. The analyzed neural networks are shallow MLPs with a power-of-ReLU activation function. The results established are of the NTK convergence type in the overparameterized regime and take the following form: Assume the width mm of the network is larger than a critical number $C$, i.e.,\n$$\n\tm > C\n$$ \nthen, with high probability with respect to the initialization, the PINN loss will converge to zero when the neural network parameters follow the gradient flow of the typical PINN loss function.\n\nThe authors analyze the dependence of the constant $C$ on the PDE order, the dimension of the computational domain, and the power of the activation function. It is proven that a high PDE order, high dimension, and a large power of the employed ReLU activation deteriorate the convergence, i.e., require a larger constant $C$ to guarantee the convergence of the gradient flow. This leads the authors to propose first-order reformulations which, as proven in the manuscript and illustrated with numerical examples, mitigate the aforementioned problems. The main novelty of the preprint lies in the quantification of the constant $C$.\n\n**General Impression:**\nThe paper is carefully written. Although the proofs are long and technical and deferred to the Appendix, the main part is an enjoyable and understandable read. However, I have a major issue with how the findings are communicated; see below in the paragraph on the weaknesses of this reply.", "questions": "See weaknesses discussion.", "limitations": "See weaknesses discussion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Well written paper, understandable to read for a broad audience albeit technical proofs.\n2. Addresses a challenging problem (training of PINNs) and tries to provide theoretical guidance on how to improve the training process.", "weaknesses": "1. There is no sharpness of the lower bound $m > C$. Thus, the conclusions the authors draw from the lower bound could potentially be wrong. This is briefly touched upon in the conclusion section of the paper, but I strongly recommend changing the presentation; see the discussion below.\n2. Achieving zero training loss is one step in a comprehensive mathematical analysis of PINNs. Other aspects, such as the discussion of the approximation error, the quadrature error, and the coercivity properties of the PINN loss function, complete the picture. I believe the authors can improve their preprint by contextualizing it better and providing pointers to the literature. Below, I have attached a list of references and briefly discussed their relevance to the overall theory.\n\n**Expansion on 1.**\n\nDrawing conclusions from a lower bound of the form $m>C$ is dangerous as for non-sharp estimates (like the present one) one can not be sure whether dependencies of $C$ on the data (PDE order, dimensionality etc) are artefacts of the proof or are truly required to obtain a lower bound. An exception would be if the lower bound is sufficiently tight, such that it is of practical value. This is not the case here: Setting $d=1$, $k=1$, $p=2$ and neglecting the log term (setting it to unity) yields\n$$\nm \\geq 2^{14} \\cdot 2^{11} \\cdot 2^{12} = 2^{37} \\approx. 10^{11}. \n$$\n\nTherefore, I strongly advise against presenting the results in a way that suggests a connection between the theoretical results (how $C$ depends on $p, k$ and $d$) and the observed optimization struggle of PINNs in the literature. Portraying results in this way leads to folklore in the community which is not backed by rigorous mathematics. At the very least, I advise the authors to clarify this early on in the introduction and before drawing conclusions from Theorem 3.2. I acknowledge that the authors discuss the limitations in the Conclusions section, but this is not where it belongs.\n\n**Expansion on 2.**\n\nThe presented results should be better embedded in the existing literature. As a matter of fact, this will strengthen the authors points, especially with respect to the PDE reformulation in first-order systems. I recommend the following articles:\n\n- [1] The article discusses error analysis/coercivity estimates for PINNs with sharp coercivity estimates. See for instance equation (3.10) in [1]. In this equation $L$ is the loss and $\\eta$ quantifies the mismatch between discrete loss (analyzed in the authors preprint) and population loss, which behaves according to Monte-Carlo integration and can be analyzed by Rademacher complexity arguments, as for example illustrated in [2]. Furthermore, the validity of equation 3.10 in [1] relies on coercivity properties of the PDE at hand and is illustrated for a number of linear PDEs in [1]. With the aforementioned articles, the errors analysis of PINNs is more comprehensive and I strongly believe readers should be pointed in this direction when reading the authors preprint.\n\n- The benefit of first-order reformulations of PINN type losses is well-known in the literature, also for different reasons than the ones the authors point out. For example [3] reports better results for jumping material coefficients and shows that strong formulations will not work at all in this case. Another aspect is treated in [1]. It is shown that first order reformulations lead to improved convergence rates, compare the error estimates for  equation to the ones of Poisson equation. To achieve $H^{1/2}$ convergence for Poisson, one needs approximation in $H^2$, for $H^{1/2}$ convergence for Darcy (which can be viewed as a first order reformulation of Poisson) one needs approximation in $H^1$ for the pressure and $H(\\operatorname{div})$ for the velocity, which leads to improved rates.\n\nOverall, I think the paper merits publication if the weaknesses are addressed and I am willing to raise my score in this case.\n\nReferences:\n\n[1] https://arxiv.org/abs/2311.00529\n\n[2] https://arxiv.org/pdf/2308.16429\n\n[3] https://www.sciencedirect.com/science/article/abs/pii/S0021999120304812", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. There is no sharpness of the lower bound $m>C$. Thus, ...\n\n**R**: We acknowledge the reviewer’s concern regarding the sharpness of the bound we presented in our paper. We concede that we have not demonstrated the sharpness of the bound in our theorems. However, it is important to note that the leading term of the bound we derived is based on conditions necessary for the Gram matrix to be positive definite, which is a crucial property of the Gram matrix for ensuring the convergence of the gradient flow (GF) to a global optimizer. \nSince the Gram matrix is defined by the PDE loss and the network structure, factors such as the PDE order of the power of the ReLU activation inherently affect the positive definiteness of the Gram matrix.\n\nTherefore, even though the bound we provide may not be sharp, we believe it can still offer valuable insights into how reducing the order and power improves convergence. It underscores how variations in PDE order and ReLU power affect convergence, thus providing the first theoretical demonstration of the relationship between the PDE order and the convergence of PINN, which has been empirically observed frequently. We believe this is an important step in understanding the relationship between PDE order, ReLU power, and the convergence of PINNs.\n\nHowever, there is a limitation in that we have not proven the sharpness of the bound, and as you point out, we agree that this should be clearly stated in the manuscript. We will revise the manuscript to acknowledge this limitation and adjust our argument accordingly.\n\n**Q2**. Achieving zero training loss is one step in ...\n\n-**Q2-1**. [1] The article discusses error analysis/coercivity estimates for PINNs..\n\n**R**: We would like to express our gratitude for your comprehensive and insightful feedback on our manuscript. We are particularly appreciative of the list of references and the valuable insights you have provided, which help to contextualize our work within the broader framework of mathematical analysis for PINNs.\n\nIn order to successfully employ the PINN approach for solving PDEs, it is essential to ensure that the following four conditions are met:\n* (C1) The network is capable of approximating the solution to the PDE.\n* (C2) The minimizer of the PINN population loss is the solution to the PDE.\n* (C3) The minimizer of the empirical loss approximates the minimizer of the population loss.\n* (C4) The minimizer of the empirical loss is obtainable.\nThe universal approximation theory of the network addresses C1,\nwhile the existence of exact solution to the PDE on compact domain underpin C2.\nConsequently, theoretical analyses of PINNs are primarily centered on the generalization error analysis of C3 and the optimization error analysis of C4.\n\nThe papers you suggested focus on C3, which is concerned with examining the conditions for the convergence of the generalization error. \nIn contrast, the present paper targets C4, namely the demonstration of the conditions for convergence of empirical loss.\nBesides, our result is based on the impact of the order of the governing PDE, whereas the suggested paper employed the coercivity of the PDE.\n\nDespite this, there is a paucity of theoretical research in this area. Therefore, it is imperative to analyze C3 and C4 based on the characteristics of the given PDE.\n\nAs you noted, [1] examines generalization error by capitalizing on the coercivity of the PDE operator. \nOn the other hand, Our work delves into the optimization error analysis in conjunction with the order of the PDE.\nWhile theoretical studies on PINN optimization exist, few have investigated the impact of the inherent nature of the PDE being solved. A substantial body of empirical evidence indicates that training PINNs becomes increasingly challenging as the PDE order increases, underscoring the significance of our research as a foundational step in this area.\n\nNevertheless, we concur with your point that a considerable number of steps remain in order to achieve a comprehensive understanding of the fundamental principles underlying PINNs. We believe that extending ideas in the vein of [1] could facilitate an investigation into the influence of the differential order of the PDE operator on the generalization error, thereby advancing our understanding.\n\nWe would like to reiterate our gratitude for your invaluable feedback and for sharing these references. We are eager to incorporate your suggestions to enhance our manuscript.\n\n-**Q2-2**. The benefit of first-order reformulations of PINN type losses is ...\n\n**R**: As the reviewer mentioned, both [1] and [3] examine the advantages of splitting PDEs into first-order formulations. In contrast to Reference [1], which examines the benefits from the perspective of S3, our paper focuses on how variable splitting enhances optimization in S4 by reducing the differential order included in the loss function.\nExtending our theoretical framework to analyze the impact of the variable splitting strategy on the generalization error of PINNs, as suggested by [1], represents an intriguing and significant research direction. We believe that incorporating discussions on these references would enrich our paper. Therefore, we plan to include them and the additional discussions in the revised version of our manuscript. We are confident that these enhancements will contribute to the depth and breadth of our work.\n\n\nThank you again for your invaluable feedback and suggestions. We appreciate your guidance in improving our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UjjZsDwQm", "reviewer": "Reviewer_BvZt", "summary": "- Building upon the work of Gao et al. [22], the authors extend the analysis of the gradient flow (GF) of PINNs (two-layer MLP is assumed) to general $k^\\mathrm{th}$ order partial differential equations (PDEs) and the $p^\\mathrm{th}$ power of Rectified Linear Unit (ReLU) activation function with general $p$.\n- The authors achieve tighter bounds than those obtained by Gao et al [22].\n- The width of the network necessary for the convergence of GF increases exponentially with the power $p$ of $\\mathrm{ReLU}^p$. \n- The optimal power $p$ is determined by the order $k$ of the governing PDE; $p = k +1$.\n- The GF convergence of PINNs deteriorates with increasing dimensions.\n- To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy, a conventional order-reduction approach.\n\n- An order reduction through variable splitting is proposed based on the theoretical findings above.\n\nOverall, I enjoyed reading the manuscript; however, it would benefit from some revisions.", "questions": "- Can the theoretical analysis given in this paper generalize to other activation functions, e.g., leaky $\\mathrm{ReLU}^p$, softplus, and GeLU, sine, etc.?\n\n- Can the theoretical analysis given in this paper generalize beyond two-layer networks?\n\n- How can we extend this work to non-gradient-flow-based optimization problems, which are more prevalent and practical, e.g., random collocation points with resampling (i.e., SGD-like training)?\n\n- To what extent theoretical predictions deviate from its corresponding experimental results when SGD-like training is used?\n\n- As mentioned above, it is shown that theoretical predictions of gradient flow deviate from experimental results as $t \\rightarrow \\infty$ even when learning rates are impractically small as shown in [Miyagawa, 2022 (https://openreview.net/forum?id=qq84D17BPu)]. How can we alleviate this issue, or is this possibly not a problem in the present paper?\n\n- (Footnote 3 on page 5) Is there any other evidence or related work that smoother activation functions require larger networks for convergence? The opposite, i.e., \"smoother activation functions help convergence and *reduce* the required size of width,\" would also sound valid in the context of general DNN training. I would greatly appreciate if the authors discuss this point further.\n  - Inequality (10) and (16) seem to be *sufficient* conditions for convergence and are not necessary (please correct me if I missed something). Therefore, the intuitive explanation in footnote 3 on page 5 would be valid only in limited cases.", "limitations": "Yes (described in Section 6).", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The topic is relevant and very interesting. The theoretical relationship between PDE's orders and PINN's convergence has been elusive but is a practically important. A challenge comes from the fact that PINN's objective contains the derivatives of PINN's outputs w.r.t. its inputs (Gao et al. [22] is one of pioneering works in this sense).\n\n- Theoretical contributions include:\n  - Building upon the work of Gao et al., the authors extend the analysis of the GF of PINNs, composed of two-layer MLPs, to general $k^\\mathrm{th}$ order PDEs and the $p^\\mathrm{th}$ power of ReLU activation function with general $p$. The authors achieve tighter bounds than those obtained by Gao et al.\n  - The width of the network necessary for the convergence of GF increases exponentially with the power $p$ of $\\mathrm{ReLU}^p$. \n  - The optimal power $p$ is determined by the order $k$ of the governing PDE; $p = k +1$.\n  - The GF convergence of PINNs deteriorates with increasing dimensions.\n  - To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy, a conventional order-reduction approach.\n\n- This paper is easy to follow.\n\n- The code is submitted for reproduction, which is important because studies on PINNs are often hard to reproduce due to their strong dependence on random seeds. However, the submitted code does not work, unfortunately (see Weaknesses below).", "weaknesses": "### Major comments\n\n- The most critical concern is that experiments were conducted using Adam, not GD; thus, Section 5 does not validate the theoretical results, which are based on GF. \n  - While training PINNs with GD is challenging (as noted in line 320), I strongly recommend including experimental results using GD, which would significantly enhance the paper's quality. Although some preceding papers have used Adam or other irrelevant optimizers to validate their theoretical results built upon GF, such experiments are, in my opinion, obviously ill-advised.\n  - A continuous-time limit of Adam is given in [Malladi et al., 2022 (https://arxiv.org/abs/2205.10287)], which is, of course, different from GF.\n\n- While GF is easy to handle in theory, GF's underlying assumptions are too restrictive in practice. GF is be a good approximation only when (1) PINNs are trained without random collocation points or are trained on random collocation points without resampling, (2) full-batch training is assumed, and (3) sufficiently small learning rates are used. There three conditions are too restrictive in view of current PINN experiments in the literature. Convergence analyses of DNNs are often criticized for their gaps between theory and practice. I greatly appreciate if these concerns are resolved.\n  - Condition (1) is mentioned in line 358. I would like to ask a question about this point, together with Condition (2), in Questions below.\n  - Condition (3) is mentioned in line 361. It is shown in [Miyagawa, 2022 (https://openreview.net/forum?id=qq84D17BPu)] that theoretical predictions of gradient flow deviate from experimental results as $t \\rightarrow \\infty$ even when learning rates are impractically small. This work would be very helpful and significantly extend the present work, adapting it to practical experimental settings. Please see Questions below.\n  \n\n### Code\nOverall, I would recommend that the authors follow the official code submission guidelines and templates (https://github.com/paperswithcode/releasing-research-code).\n\n- The 'utils' module is missing in the submitted codes, and they do not work.\n\n- Adding requirements.txt would be recommended.\n\n\n### Minor comment\n\n- $\\mathrm{ReLU}^p$ is called the rectified power unit (RePU) in [Bokati et al. (https://scholarworks.utep.edu/cgi/viewcontent.cgi?article=2717&context=cs_techrep)]. Please see also the references therein. I would recommend citing them in the submitted paper.\n\n\n\n\n### Typos\n- Line 154: Impact of PDE order on convergence of PINNs -> Impact of PDE Order on Convergence of PINNs\n- Line 203: $\\delta << 1$ -> $\\delta \\ll 1$\n- Line 372: keller-segel -> Keller-Segel\n- Line 373: da vinci-euler-bernoulli -> da Vinch-Euler-Bernoulli\n- Line 414: pinns -> PINNs\n- Line 462: i -> I\n- Line 507: Dgm -> DGM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In order to provide a concise response to theoretical questions, we structured our proof:\n* (S1) Determine the condition for the positive definiteness(PD) of the initial Gram matrix.\n* (S2) Establish an upper bound for the initial loss in terms of a polynomial of the initial weights.\n* (S3) Identify a radius of an open ball within which the Gram matrix preserves PD.\n* (S4) Compute the GF and demonstrate that the flow converges within the ball.\n\n**Q1**. The most critical concern is ...  \n* I strongly recommend ...  \n**R**: We agree with the concern that the experiments in Section 5 do not fully validate our theorems as they use Adam.  \nIn response to the feedback, we conducted experiments using the GD. As achieving convergence with GD for higher-order PDEs is challenging, we compare for the heat equation in Eq. (18). Furthermore, in accordance with the request of Reviewer M6m4, experiments were conducted on the convection-diffusion equation. We train models via GD with a learning rate(lr) of 1e-1. PINNs on the convection-diffusion equation employ a reduced lr of 1e-2 as it diverges with a lr of 1e-1. Figures A3 and A4 of the attachment present fast and remarkably stable convergence of VS-PINNs with smaller variance.  \nMoreover, we have designed experiments with GD that can clearly support our theory. A detailed description and discussion can be found in the common response.  \nWe believe these provide further validation of our findings. The results of the experiments in Section 5 may suggest that results to those with GD can be achieved with Adam although we have not proven.\n* A limit of Adam ...  \n**R**: We append experiments using GD.\n* extend to non-GF-based ...  \n**R**: We coud apply our theory to SGD-like optimization. In the context of stochastic approximation theory (SAT), errors on a gradient originate from the randomness is interpreted as noise when estimating the GF. Following the SAT, the noisy GF converges to a fixed point of the clean GF. Consequently, our theory could be extended.  \nHowever, we acknowledge that S1 is not applicable when collocation points are resampled. In S1, we use the fact that gradients at each collocation point are linearly independent if width is sufficiently large, which does not hold as there are infinitely many collocation points in distribution. Assuming S1, we expect that the S2-S4 are applicable, given that expectation forms and uniformly bounded losses are employed.\n* As mentioned above, ...  \n**R**: We acknowledge that GF and GD exhibit different dynamics. However, we believe our theory can be adapted to GD, given backward analysis, including [Miyagawa, 2022].  \nTheorem 5.1 and its corollaries in [Miyagawa] show that GF and GD dynamics differ in the limit $t\\rightarrow\\infty$, assuming a scale-invariant layer is present. Our theory, however, is based on MLPs without normalization layers, and thus does not meet the condition in the theorem.  \nInstead, we could apply Theorem 3.3 from [Miyagawa], which states that the GD dynamics follows GF with a counter term. Since the term is also a polynomial, we anticipate that S2-S4 can be adjusted. In S1, we expect our induction on a maximum degree can be adapted because the leading term occurs only in the counter term. Therefore, we expect to prove the convergence of GD based on the backward analysis.  \n\n**Q2**. I would recommend that ...  \n**R**: We apologize and thank you for pointing this out. We will rectify it by including the 'utils' module and providing a requirements.txt file to specify the necessary dependencies. We regret any inconvenience this may have caused.  \n**Q3**. ReLU^p is called the RePU  \n**R**: We appreciate your recommendation to cite the references related to the RePU. We will include these citations.  \n**Q4**. Generalize to other activation?  \n**R**: S1 effectively exploits the non-smooth nature of the RePU. This approach can be expanded to include other non-smooth activation functions like Leaky RePU. Additionally, it seems feasible to bound the loss function using a polynomial of the parameters. Therefore, extending the S2-S4 to this type of activation function seems plausible.  \nHowever, our approach in S1 is not applicable for smooth activations and it is difficult to establish the PD of the Gram matrix. This leads some studies to assume the PD of the Gram matrix and analyze the convergence of the optimization flow. Assuming S1, our results  would extend to activation functions whose derivatives are polynomially bounded.  \n**Q5**. Generalize beyond two-layer?  \n**R**: S2-S4 of our theory would remain feasible as the composition of polynomials is a polynomial. However, the non-smooth point can be entangled, making the applicability of S1 less straightforward.  \n**Q6**. Is there any other evidence that ...  \n**R**: There are theoretical papers that analyze the impact of activation in terms of optimization convergence. \nFor instance, [Panigrahi, 2020] computes the eigenvalue of the Gram matrix for both non-smooth and smooth activations. \nThe authors provided evidence indicating that when data span a low dimensional space, the smallest eigenvalue of the Gram matrix can be small leading to slow training. \nIt is also demonstrated that the smallest eigenvalue of the Gram matrix associated with non-smooth activations is larger than that of smooth activations under minimal assumptions on data. Consequently, the findings imply that training will be more rapid for non-smooth activation.  \n**Q7**. Seem to be sufficient condition  \n**R**: Although they are sufficient conditions, the leading terms originate from the PD of the Gram matrices, which is a prerequisite for convergence. Given that the PD of the Gram matrices inherently depends on  the PDE and p, we contend that the insights regarding the detrimental impact of large p could be derived from the inequalities.  \n**References**\nPanigrahi et al., Effect of Activation Functions on the Training of Overparametrized Neural Nets, ICLR, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to All Reviewers**

We sincerely appreciate all the reviewers for their invaluable comments, recommendations, and suggestions. The opinions of the reviewers have been  carefully considered and responding to their questions has enhanced the paper. We first introduce additional experiments that are commonly required to address the comments from the reviewers inw1 and BvZt. Subsequently, we address the individual responses to each reviewer below. We also attach a supplementary file for Figures and a Table. Hopefully, the replies could adequately address all the concerns.

**Common Response**:
In response to the feedback from reviewer inw1, we designed and conducted additional experiments that better demonstrate the theoretical results of our paper. The objective of this experiment was to validate the effect of power and PDE order on the width size required for convergence, which represents the primary result of our theoretical findings. The experimental results obtained were found to be consistent with the theoretical results, which we believe addresses the concerns of the reviewers.
* *(Experimental setup)*: To demonstrate the influence of $p$, $k$ on $m$, we trained networks with various $m$, ranging from $10^2$ to $10^6$, for each $p$ and $k$ and investigated their convergence under GD optimization. Networks were trained by GD with a learning rate of 1e-8. The utilization of this small learning rate facilitates the observation of the preliminary convergence behavior without causing training instability or divergence. Given the considerable number of steps required for full training, we did not undertake full training.
Training collocation points were set to uniform grids: 400 in the domain and 100 on each boundary of the domain.\
To examine the influence of the PDE order $k$ on the width required for convergence, we additionally considered a Poisson equation ($k=2$) in conjunction with the bi-harmonic equation ($k=4$) presented in Eq. (508) of the submitted manuscript. The Poisson equation is subject to a homogeneous Dirichlet boundary condition in accordance with Eq. (508) and the source function is set to yield the same solution as that of Eq. (508). Other training configurations are the same as the bi-harmonic equation represented in Appendix D. 

The results are presented in Figure A1 of the attachment, which illustrate the loss behavior for networks with various widths $m$ for each $p$.
The results clearly support our theoretical finding that the width should increase to ensure convergence as $p$ increases. Moreover, a narrower network would converge when solving a lower-order equation (Poisson), suggesting that a higher-order PDE (bi-harmonic) requires a wider network.
In summary, the results show that the width needed for convergence grows with p and the PDE order. This validates our paper's theoretical results and shows the importance of our study in analyzing how the PDE order and the activation power affect convergence of PINNs.


**Supplementary file &darr;**

Meta-review: The paper studies the behavior of neural network based solvers for high order PDEs. The authors proposed variable splitting approach to reduce the order of the PDE to facilitate the solutions. Overall the reviewers find the work interesting and solid, and all their concerns are addressed over the discussion period. Hence the meta-reviewer recommends acceptance of the paper. 

For the final version, it would be worthwhile to add some discussion on the treatment of boundary conditions, as it is not obvious how to impose correct boundary conditions for the new variable introduced in the splitting.
Decision: Accept (poster)