Title: How does PDE order affect the convergence of PINNs?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Physics-Informed Neural Networks

Variable Splitting

Higher-order PDEs

Gradient Flow

Convergence
Abstract: This paper analyzes the inverse relationship between the order of partial differential equations (PDEs) and the convergence of gradient descent in physics-informed neural networks (PINNs) with the power of ReLU activation. The integration of the PDE into a loss function endows PINNs with a distinctive feature to require computing derivatives of model up to the PDE order. Although it has been empirically observed that PINNs encounter difficulties in convergence when dealing with high-order or high-dimensional PDEs, a comprehensive theoretical understanding of this issue remains elusive. This paper offers theoretical support for this pathological behavior by demonstrating that the gradient flow converges in a lower probability when the PDE order is higher. In addition, we show that PINNs struggle to address high-dimensional problems because the influence of dimensionality on convergence is exacerbated with increasing PDE order. To address the pathology, we use the insights garnered to consider variable splitting that decomposes the high-order PDE into a system of lower-order PDEs. We prove that by reducing the differential order, the gradient flow of variable splitting is more likely to converge to the global optimum. Furthermore, we present numerical experiments in support of our theoretical claims.

How does PDE order affect the convergence of PINNs?

Abstract

This paper analyzes the inverse relationship between the order of partial differential equations (PDEs) and the convergence of gradient descent in physics-informed neural networks (PINNs) with the power of ReLU activation. The integration of the PDE into a loss function endows PINNs with a distinctive feature to require computing derivatives of model up to the PDE order. Although it has been empirically observed that PINNs encounter difficulties in convergence when dealing with high-order or high-dimensional PDEs, a comprehensive theoretical understanding of this issue remains elusive. This paper offers theoretical support for this pathological behavior by demonstrating that the gradient flow converges in a lower probability when the PDE order is higher. In addition, we show that PINNs struggle to address high-dimensional problems because the influence of dimensionality on convergence is exacerbated with increasing PDE order. To address the pathology, we use the insights garnered to consider variable splitting that decomposes the high-order PDE into a system of lower-order PDEs. We prove that by reducing the differential order, the gradient flow of variable splitting is more likely to converge to the global optimum. Furthermore, we present numerical experiments in support of our theoretical claims.

1 Introduction

Understanding of partial differential equations (PDEs) is fundamental in describing diverse phenomena in science and engineering, including fluid dynamics [60, 17], weather prediction [51], disease progression [3, 46], and quantum mechanics [20, 8]. This underscores the imperative necessity for the effective acquisition of their solutions. Given that analytically solving PDEs is often infeasible or even impossible for numerous practical scenarios due to their complexity, numerical methodologies play a pivotal role in approximating solutions to PDEs, enabling researchers and engineers to address real-world problems effectively.

The advent of deep learning has led to a surge in attempts to leverage it to solve PDEs [59, 42, 30]. Among these, physics-informed neural networks (PINNs) [39, 21, 38, 58] stand out as a prominent methodology. Coupled with the automatic differentiation technique [7], they integrate the residuals of PDEs and boundary conditions into the loss function, thereby enforcing the approximation of solutions using artificial neural networks. This distinctive incorporation of PDEs into the loss function introduces partial differential operators in calculating the loss, distinguishing PINNs from conventional deep learning models. Renowned for their accessibility and versatility in being capable of easily handling arbitrary PDEs and being mesh-free, PINNs have garnered significant attention and demonstrated promising outcomes across various fields [13, 28, 2, 65].

Despite their potential, PINNs frequently encounter difficulties in accurately approximating solutions, particularly when the governing PDE contains high-order derivatives [48, 33]. They also exhibit sensitivity to increasing dimensions [33]. These challenges impede the practicality of PINNs due to the pervasiveness of high-order or high-dimensional PDEs in numerous physical and engineering descriptions, such as control problems [22, 57], finance [5, 53], phase separation [12, 27], and mechanical engineering [4, 32]. Several studies have indicated that neural network architectures possess sufficient expressive power to approximate solutions [31, 41]. However, it has been purported that the inferior performance may be attributed to the difficulty in optimizing PINNs, which arises from including the PDE in the loss function [37, 61, 62]. Despite the widespread use of PINNs, a rigorous mathematical understanding of these pathological behaviors of PINNs has been lacking.

In this paper, we endeavor to provide a mathematical understanding of the pathological behaviors of PINNs by analyzing the convergence of their gradient flow (GF), which reveals a profound sensitivity of the GF convergence with respect to the PDE order and the power of the activation. Building upon the work of Gao et al. [25], we extend the analysis of the GF of PINNs, composed of two-layer multilayer perceptrons (MLPs), to general k th-order PDEs and the p -th power of Rectified Linear Unit (ReLU) activation function with general p . We achieve tighter bounds than those obtained by Gao et al., shedding light on the underlying causes of the pathological behaviors of PINNs. Our theoretical findings demonstrate that the width size of the network necessary for the convergence of the GF increases exponentially with the power p of ReLU p activation. Furthermore, our results indicate that the optimal power p is determined by the order k of the governing PDE, specifically to be k +1 . We also find that the PDE order impedes the convergence of GF, where this negative impact of the PDE order stems from incorporating the PDE into the PINN loss function, which necessitates network differentiation up to the order of the PDE. Moreover, our theoretical investigation unveils that the GF convergence of PINNs also deteriorates with increasing dimensions, and the differential operators included in the PINN loss further exacerbate the sensitivity of PINNs to dimensionality. This elucidates why PINNs are relatively sensitive to dimensionality compared to conventional deep learning models that do not involve differentiation in the loss function.

To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy [54, 55, 6], which represents derivatives of the solution as additional auxiliary variables. The key point of variable splitting is that learning a high-order PDE boils down to learning a system of lower-order PDEs. Reducing the order of derivatives included in the loss function, the strategy alleviates the difficulties associated with the PDE order. It further enables to utilize more general ReLU p activation with lower power p than PINNs. The lower differential orders that the network computes, the more likely it is that the GF will converge, so the most suitable one among the various constructions of the variable splitting method is the finest splitting, which separates all the derivatives into auxiliary variables and reformulates the PDE into a system of first-order PDEs. This strategy results in a loss function comprising only first-order derivatives, and the efficacy of this finest variable splitting would be magnified as the order of the governing PDE or dimension increases. Therefore, the finest splitting approach would exhibit a pronounced discrepancy from the vanilla PINNs for high-order PDEs. Moreover, a reduction in the differential orders enhances the resilience of the model with respect to dimensionality. Finally, we present numerical experiments to verify our theoretical findings and validate the effectiveness of the variable splitting.

1.1 Related Work

Characterization of Gradient Descent for PINNs As significant issues have been identified within physics-informed machine learning, numerous mathematical studies have been conducted to elucidate the behavior of PINNs. While studies have been mainly dedicated to examining the generalization capacity of PINNs [19, 49, 23], there has also been work on understanding the difficulty of optimization, which is believed to be the primary source of failure for PINNs. Wang et al. [61] found that PINNs exhibit stiff gradient flow dynamics, resulting in imbalanced gradients during training. Ryck et al. [18] characterized the rate of convergence in terms of the conditioning of an operator and suggested that the difficulty of training PINNs is closely related to the conditioning of the differential operators in the governing PDEs. Another work [62] utilized the neural tangent kernel (NTK) theory to indicate that spectral biases and discrepancies between convergence rates of various loss components can lead to training instabilities. Global convergence properties of PINNs for second-order linear PDEs have also been studied within the NTK regime [34] and using the Rademacher complexity [47]. Most closely related to this paper, Gao et al. [25] demonstrated the

convergence of the gradient descent for two-layer PINNs. However, their discussion is limited to second-order linear PDEs. We extend the analysis to general k th-order linear PDEs and p -th power of activation functions and provide tighter bounds than Gao et al.. These advances allow us to observe further the effect of the order and dimensionality of the PDE on the convergence.

Variable Splitting The method of separation of variables, which simplifies differential equations by reformulating them into a more manageable system, is a classical method for solving differential equations [9]. In particular, it has been widely employed when dealing with high-order PDEs as augmenting high-order derivatives as additional variables allows the governing equation to be decoupled into a set of lower-order PDEs that are comparatively easy to solve [16, 26, 63]. Recent endeavors have explored the integration of separable variables within the PINN approach. In this paper, we refer to this approach as variable splitting according to [54, 55, 56]. Augmented variables have been introduced to represent vorticity in the Stokes equation [6], the gradient of the solutions for solving the p -Poisson equation [54], and the eikonal equation [55]. Additionally, second-order derivatives have been separately parameterized to solve bi-harmonic equations effectively [45]. The rationale for introducing auxiliary variables in previous works is to enhance the efficiency and accuracy of PINNs, but they lack a comprehensive theoretical elucidation of its effect. A recent study [56] has theoretically analyzed variable splitting, demonstrating that while PINNs do not guarantee convergence to the PDE solution even when the loss converges to zero, variable splitting does ensure convergence to the solution for second-order linear PDEs. In this study, we analyze the impact of variable splitting for PINNs with ReLU p activation in terms of the convergence of the GF.

1.2 Main Contributions

The contribution of the paper is summarized as follows.

We analyze that the GF of PINNs with ReLU p activation converges to the global minimum for general k th-order linear PDEs. This extends the findings of Gao et al. [25] to encompass a broader range of PDEs and activations and provides an even tighter bound.

We demonstrate the inverse relation between PDE order and the GF convergence, unveiling the adverse effect of the differentials included in the PINN loss on the GF convergence.

We provide a theoretical understanding of the reasons why PINNs encounter difficulties in addressing high-dimensional problems.

We prove that the order reduction of variable splitting, which reformulates the PDEs into a system of lower-order PDEs, results in the convergence enhancement of GF.

2 Mathematical Setup

Arbitrary Order Linear PDEs We consider a general form of k th-order linear partial differential equations (PDEs) defined on a bounded domain Ω ⊂ R d (in which the temporal dimension could be a subcomponent)

where N [ u ] = ∑ | α |≤ k a α ∂ α ∂ x α u is a k th-order linear differential operator with coefficient functions a α : Ω → R for each multi-index α ∈ N d 0 , B [ u ] = ∑ | α |≤ 1 ˜ a α ∂ α ∂ x α u represents the boundary condition operator with coefficient functions ˜ a α : ∂ Ω → R , which could reflect Dirichlet, Neumann, and Robin conditions 2 , f : Ω → R is a given source function, and g : ∂ Ω → R is a given boundary function, and u : ¯ Ω → R is the unknown solution of interest.

Physics-InformedNeural Networks Physics-informed neural networks (PINNs) [58] aim to approximate the solution u of the PDE by neural networks. Following the prior work [25], we approximate the solution u by a two-layer multi-layer perceptron ϕ : R d → R of width m , defined as

where w r ∈ R d +2 , v r ∈ R , w = [ w ⊤ 0 , · · · w ⊤ m ] ⊤ ∈ R m ( d +2) v = [ v ⊤ 0 , · · · v ⊤ m ] ⊤ ∈ R m , y = [ x ⊤ 1 2 ] ⊤ ∈ R d +1 , and σ ( · ) is the activation function. For brief notations, we assume that Ω is bounded so that ∥ y ∥ 2 ≤ 1 for x ∈ Ω . We consider the case where σ is the ReLU p activation function for an integer p , which is also known as Rectified Power(RePU) activation [1, 10, 15]. As it will be clear in the context, the power p necessitates surpassing the order k of the PDE (1) to ensure that the loss function and gradient descent flow are well-defined. Therefore, our analysis is focused on scenarios where p ≥ k +1 . PINNs learn the parameters of ϕ by minimizing a composite loss function, comprising the residual of the PDE and the boundary condition of (1), which enforces the network's compliance with the governing physics. For given the training data { x i , f ( x i ) } n o i =1 ⊂ Ω × R and { ˜ x j , g (˜ x j ) } n b j =1 ⊂ ∂ Ω × R of respective sizes n o ∈ N and n b ∈ N , PINN loss function is given by

where s ( w , v ) = [ s 1 ( w , v ) · · · s n o ( w , v )] ⊤ and h ( w , v ) = [ h 1 ( w , v ) · · · h n b ( w , v )] ⊤ with

and ν > 0 is a regularization parameter that relatively balances the two components of the loss.

Gradient Flow As the limiting dynamics of the gradient descent (GD) with infinitesimal step-sizes [40], gradient flow (GF) is continuous time dynamics that starts at w (0) and v (0) and evolves as

Initial weights are supposed to follow the normal and uniform distributions, w (0) ∼ N (0 , I m ) and v ∼ U ( {-1 , 1 } ) , respectively 3 . GF can be regarded as a continuous-time analog of GD and is frequently employed to comprehend the behavior of GD optimization algorithm in the limit. By the chain rule in conjunction with (6), the following characterizes how the loss function evolves during training by gradient descent:

where G and ˜ G are Gram matrices for the dynamics, defined by

We are interested in analyzing the effect of the PDE order on the convergence of the PINN loss, which evolved in accordance with the dynamics (7), to the global minimum zero.

3 Impact of PDE Order on Convergence of PINNs

Despite the demonstrated promise and versatility of PINNs in addressing a wide range of problems [29, 44, 36], they often encounter difficulties in constructing an accurate approximation to the desired solution of PDEs, particularly with high-order PDEs. Moreover, in contrast to the confirmed efficacy of neural networks in modeling high-dimensional data such as images and text, the exploration of PINNs for high-dimensional PDEs has been apparently limited. While neural network architectures possess sufficient expressive power to approximate solutions [11], inferior performance has been attributed to the difficulty in optimization in practice [61, 62]. Additionally, it has been postulated that the optimization difficulty may stem from the partial differential operators included in the loss function [37, 48, 33]. Nevertheless, despite the significant challenge posed by these pathological phenomena, there remains a paucity of theoretical understanding of them.

In this section, we theoretically elucidate these pathological phenomena by studying the convergence condition of GF (7) of PINN loss (3). Specifically, we provide a width condition for (7) to converge to global optimum in terms of order k , dimension d , and the power p of ReLU activation. Analyzing how those factors are related to the convergence condition, we explain why optimizing PINNs is harder when the order or degree is higher.

Following [24] and [25], we first prove the positive definiteness of the limiting Gram matrix of PINNs for general k th-order linear PDE and p without any further strict assumption other than p > k .

Proposition 3.1 (Special Case) . The limiting Gram matrix G ∞ v = E w , v [ G v ( w , v )] is strictly positive definite and independent of m .

This is a special case of the general state in Proposition C.3 with L = 0 , and the proof for the general case is provided in Appendix C. We denote the smallest eigenvalue of G ∞ v by λ 0 > 0 . The following presents our main theorem in this section, the requisite width size m for the GF of PINN loss to converge to the global minimum with high probability. The result demonstrates that the required width grows exponentially as the PDE order k and the dimension of the domain d increase.

Theorem 3.2 (Special Case) . There exists a constant C , independent of d , k , and p , such that for any δ < < 1 , if

then with probability of at least 1 -δ over the initialization, we have

It is a special case of Theorem 4.3 with L = 0 , the proof of which can be found in Appendix C.1. It extends, inspired by [25], the convergence of the GF of PINNs of second-order linear PDEs to k th-order linear PDEs and the general p -th power of ReLU. It states that even in these general settings, the GF of PINNs converges to the global minimum with a high probability when the width of the network is sufficiently large. Moreover, we obtain a polylogarithmic bound ( log 1 δ ) 4 p , which is much tighter than polynomial bound δ -3 in [25] for p = 3 . These improvements permit the derivation of the following valuable explanations for the deficiencies observed when optimizing PINNs.

Optimal Power of ReLU Function in Training Theorem 3.2 sheds light on the suitable choice of activation function for PINNs. In the training process, the activation function plays an important role. However, there are no clues as to which activation function is favorable to the given optimization process. Especially in the case of PINNs, it depends heavily on the PDE at hand. Despite its pervasive use in deep learning due to its numerous advantages and performance benefits, the ReLU activation function is not admissible in the PINN framework, which necessitates the activation function to provide high-order derivatives for optimizing PDE-based constraints. Instead, PINNs harness the p -th power of ReLU as the activation function. It is apparent that p must satisfy p ≥ k +1 for the PINN loss and gradient descent to be computed. Theorem 3.2 indicates that the smaller p is, the more likely the gradient descent will converge; that is, it is most optimal 4 to adjust p to k +1 regarding the training process.

Understanding Difficulty in High-order PDEs A significant observation of Theorem 3.2 is that it provides a theoretical understanding of why PINNs struggle with high-order PDEs. From (10), we can see that the bound increases exponentially with the order of the PDE. Moreover, for the GF of PINNs to converge with high probability, that is, δ ≪ 1 , a small increment of the power p would contribute to non-negligible degradation in the convergence, which could ultimately prevent the network from reaching a minimizer of the loss. Hence, given that k determines the admissible p by p ≥ k +1 , the order k of PDE primarily influences the convergence of PINNs and increasing the exponential term in (10).

Understanding Difficulty in High-dimensional Problems The above theorem, which shows that the lower bound of m depends on the exponential of d , explains why PINNs cannot completely combat the curse of dimensionality. As PINNs are regarded as a versatile method capable of being mesh-free, they have been expected to be free from the curse of dimensionality [58]. However, the GF of PINNs becomes harder to converge as d increases, requiring the network to be wider. Furthermore, it can be observed that the magnitude of change in d is amplified with respect to the exponent of k . This explains why PINNs are relatively sensitive to increasing dimensionality in comparison to other deep learning models whose loss functions do not contain derivatives. In other words, the presence of derivatives in the loss makes PINNs sensitive to changes in dimensionality, and the larger k is, the more difficult PINNs are for high-dimensionality.

Combining all crucial observations from our main theorem, we believe that the impact of the PDE order is one of the primary underlying reasons why PINNs often fail to minimize their loss. In light of this theoretical evidence, the next section describes a variable splitting strategy that addresses these pathologies by properly reducing the differential order in the PINN loss function.

4 Order Reduction through Variable Splitting

The previous section indicates that the PDE order k significantly affects the width requirement for the GF to converge. Concurrently, for k th-order PDEs, it is necessary to increase the ReLU activation to at least the k +1 power in order to ensure a well-defined GF for the PINN loss. Consequently, lowering k could potentially lead to better convergence of the GF. In this section, we introduce variable splitting strategy to decrease the differential order by reformulating the given PDE into a system of lower-order PDEs. We then extend Theorem 3.2 to a more general form in Theorem 4.3.

4.1 Variable Splitting

The concept of variable splitting [54, 55, 56] is to rewrite a higher-order PDE into a lower-order system, after which the PINN approach is applied to the system. A crucial aspect of the success of such methods is the reduction of the derivative order present in the training loss function.

Augment Variables For L ≥ 0 and increasing integers 0 = ξ 0 < ξ 1 < · · · < ξ L +1 = k , variable splitting augment the derivatives of the solution ∂ ξ 1 ∂ x ξ 1 u, . . . , ∂ ξ L ∂ x ξ L u as additional auxiliary variables ϕ 1 , . . . , ϕ L , respectively. For notational simplicity, we abbreviate the integer set { 1 , · · · , m } for a positive integer m by [ m ] . For ℓ ∈ [ L ] , each term in ∂ ξ ℓ ∂ x ξ ℓ corresponds to ∂ ξ ℓ ∂x α 1 1 ··· ∂x α d d u for a multi-index α = ( α 1 , . . . , α d ) ∈ N d 0 with the size | α | = ∑ d i =1 α i = n . Therefore, ϕ ℓ is a vectorvalued function of size | I ξ ℓ | for the index set I defined in (18). We denote the component of ϕ ℓ that corresponds to ∂ ξ ℓ ∂ x α by ( ϕ ℓ ) α .

Reformulate PDE into Lower-order System By replacing each of the differential terms ∂ ξ ℓ ∂ x ξ ℓ u with the corresponding auxiliary variables ϕ ℓ , the differential operator N in (1) can be rewritten as:

not that a network can be easily trained. Indeed, our result leads us to the opposite conclusion that large p could be detrimental to convergence from the optimization perspective. In this paper, we refer to 'optimal' as the sense of being likely trained under the mildest condition rather than approximating the solution with the smallest error.

for some coefficient functions ˆ a ℓ,α,β : Ω → R and ∆ ξ ℓ = ξ ℓ -ξ ℓ -1 . Since ϕ ℓ represents a function that differentiates the PDE solution ∆ ξ ℓ -times more than ϕ ℓ -1 , the components of two consecutive variables ϕ ℓ -1 and ϕ ℓ are governed by

From these, the PDE (1) can be identically reformulated by the system of lower-order PDEs:

It is of paramount importance to note that the maximum differential order of this system of PDEs is the highest difference of derivative order between consecutive auxiliary variables | ξ | = max { ∆ ξ ℓ : ℓ ∈ [ L +1] } , which is less than k . This aspect gives rise to notable ramifications in our analysis of VS-PINNs, which will be discussed in the next subsection.

Variable Splitting for PINNs In this paper, we consider the parameterization of all variables ϕ ℓ with two-layer MLPs with ReLU p activation function, in a manner analogous to that described in Section 2 for PINNs. The weights in the first and second layers of ϕ ℓ are denoted by w ℓ and v ℓ , respectively.We use w = [ w ⊤ 1 · · · w ⊤ L ] ⊤ and v = [ v ⊤ 1 · · · v ⊤ L ] ⊤ to refer to the respective collections of all weights. Similar to PINNs, Variable Splitting for PINNs (VS-PINNs) employ the linear sum of penalized residuals of each term of the induced system of PDEs (14) as the training loss:

where ν, ν 1 , . . . , ν L are regularization parameters. As the GF of L PINN is characterized by Gram matrices G w and G v induced from the gradients of the residuals of each term in (1), the GF of L V S PINN is characterized by Gram matrices ̂ G w and ̂ G v , which is induced from the gradients of the residuals of each term in (14). Appendix A gives more details for ̂ G w and ̂ G v .

Remark 4.1 . In order for high-order PDEs with k > 2 to be well-posed, it is necessary to have more boundary conditions than those defined by the boundary operator B in (1). Although our analysis concentrated on B that reflect only up to first-order derivatives for the sake of simplicity, our theory can also be applied to more general boundary conditions. Furthermore, the high-order boundary conditions B can also be reformulated using the auxiliary variables used for ˆ N . In that case, relations (13) should hold on the boundary x ∈ ∂ Ω . As the reduced system (14) with reformulated boundary condition is equivalent to (1), instability issues were not observed in our numerical experiments even in the absence of artificial boundary conditions on the auxiliary variables unlike to grid-based conventional numerical schemes.

4.2 Analysis

A key advantage of VS-PINNs is that the derivative order of the induced system of PDEs (14) is | ξ | , which is lower than that of the original PDE (1). We prove its effectiveness in this section. As analogous to PINNs, we begin by proving the positive definiteness of the limiting Gram matrix, providing its proof in Appendix C.

Proposition 4.2 (General Case) . The limiting Gram matrix ̂ G ∞ v = E w , v [ ̂ G v ( w , v ) ] is strictly positive definite and independent of m .

We denote the smallest eigenvalue of ̂ G ∞ v by λ 0 > 0 . We now present our main theorem, which demonstrates the profound impact of order reduction in variable splitting. The proof of the following theorem can be found in C.1.

Theorem 4.3 (General Case) . There exists a constant C , independent of d , k , | ξ | , and p , such that for any δ < < 1 , if

then with probability of at least 1 -δ over the initialization, we have

The right-hand-side of (16) grows exponentially with respect to d , k , and p , thereby indicating the substantial influence of these factors on the convergence of VS-PINNs, including PINNs as a specific case ( L = 0 ). This analysis reveals several significant advantages of VS-PINNs:

Improved Convergence: VS-PINNs are more likely to converge to the global optimum than PINNs due to the reduction in the derivative order | ξ | < k . This also relaxes the condition on p from p ≥ k +1 to p ≥ | ξ | +1 . As previously discussed in Section 3, the optimal value of p is | ξ | +1 . Given that 6 ( | ξ | +1) is an exponent of log (1 /δ ) , the most dominant term, reducing the order from k to | ξ | leads to an immense improvement. There is another noteworthy observation we can see here. Given a k th-order PDE, there are numerous possible partitions ξ that could be employed to decompose it to a system of lower-order PDEs. Consequently, there are a many of potential VS-PINNs that could be constructed. The aforementioned result indicates which of these is the most effective. As the convergence improves dramatically with a reduction in the derivative order, the optimal approach for splitting variables among various ways is to separate the given PDE into a system of first-order PDEs by parameterizing all derivatives of the solution as auxiliary variables. In other words, the finest splitting with ξ 0 = 0 , ξ 1 = 1 , . . . , ξ k -1 = k -1 would be the most effective in terms of the convergence of GF, as the differential order | ξ | is reduced the most to 1. Taken all together, the most optimal VS-PINNs that reduce the PDE order k to 1 will markedly enhance the convergence of GF.

Reduced Dimensional Impact: The reduction of orders in VS-PINNs enhances the resilience of the model to high-dimensionality. From Theorem 3.2, we observed the effect of d being exponentially enlarged for the PDE order k due to the k th-order partial differential operators in the loss function. It can be alleviated by VS-PINNs reducing the order, thereby easing the amplified scale to exponential of | ξ | . This indicates that VS-PINNs are more effective in combating the curse of dimensionality. Since the curse of dimensionality is a serious issue that is prevalent in various fields, including Hamilton-Jacobi-Bellman equation in control problems, Schrodinger equation in quantum physics, and Black-Scholes equation in finance, it is evident that enhancements to the robustness of VS-PINNs with respect to their dimensionality would facilitate considerable advancements in various fields.

Memory Efficiency: VS-PINNs are memory-efficient despite the presence of multiple auxiliary networks. As the order of the derivative increases, the complexity in automatic differentiation in modern deep-learning frameworks like PyTorch increases and it becomes computationally expensive [7]. Adopting the order-reduced representation in the proposed variable splitting can overcome the difficulty in calculating the high-order derivative via automatic differentiation. The loss function for the finest VS-PINNs involves only first-order derivatives, which reduces the memory usage and computational requirements. Despite the increase in the number of networks, VS-PINNs exhibit greater efficiency because memory usage and computation scale linearly with the number of networks in contrast to the exponential scaling with the order of derivatives. Table 3 in appendix demonstrates the memory reduction of VS-PINNs.

̸

Remark 4.4 . The current approach to parameterizing the ξ ℓ -th order differential operator on all axes ∇ ξ ℓ as an auxiliary variable may be suboptimal in certain cases. In a given PDE, if the order of the derivative varies significantly along the axes, that is, a α = 0 for only a few α in (1), it may be more efficient to approximate the partial derivatives using auxiliary variables separately for each axis. To illustrate, for the PDE u tt = u xxxx , it is more suitable to parameterize variables ϕ 0 ≈ u , ϕ 1 ≈ ( u t , u x ) , ϕ 2 ≈ u xx , and ϕ 3 ≈ u xxx , rather than approximating all tensors ∇ 1 ( t,x ) , ∇ 2 ( t,x ) , and ∇ 3 ( t,x ) . The theoretical framework presented in this paper is capable of addressing this scenario by constructing each ϕ ℓ to replace ∂ ξ ℓ ∂ x α for only part of α with | α | = ξ ℓ . However, we exclude it due to the intricate nature of the states and the lack of a meaningful impact on the PDE order.

1040

p=8

1040

p=7

1040

p=

1040

p=5

m=105

m=105

m=105

m=105

1030

m=104

1030

m=104

1030

m=104

1030

m=104

1020

m=103

1020

m=103

1020

m=103

1020

m=103

m=102

m=102

m=102

m=102

1010

1010

1010

1010

10

Epochs

15

20

25

30

10

Epochs

15

20

25

10

Epochs

15

20

25

10

Epochs

15

20

25

30

(a) Convergence behavior of PINNs on bi-harmonic equation

30

1040

p=8

1040

D

1040

1040

p=5

m=105

m=105

m=105

m=105

1030

m=104

1030

m=104

1030

m=104

1030

m=104

1020

m=103

1020

m=103

1020

m=103

1020

m=103

m=102

m=102

m=102

m=102

1010

1010

1010

1010

10

Epochs

15

20

25

30

10

Epochs

15

20

25

10

Epochs

15

20

25

10

Epochs

15

20

25

(b) Convergence behavior of PINNs on Poisson equation

Remark 4.5 . Although the shaprness of the bound in Theorem 4.3 is open, it is important to note that the leading term of the bound is based on conditions necessary for the Gram matrix to be positive definite, which is a crucial property of the Gram matrix for ensuring the convergence of the GF to a global optimizer. Since the Gram matrix is defined by the PDE loss and the network structure, we believe it can still provide valuable insight into how order and power affect convergence.

5 Experiments

This section presents experimental results that validate the theory. Throughout numerical experiments, two-layer MLPs with ReLU p activation function were utilized in order to align with our theoretical framework. Throughout all experiments, the training collocation points consists of uniform grid and regularization parameters are set to ν 1 , . . . , ν L = 1 and ν = 10 . We implement all numerical experiments on a single NVIDIA RTX 3090 GPU. Experimental details are provided in Appendix D.

Convergence behavior of PINNs To investigate the influence of the activation order p and the PDE order k on the width m required for convergence, we examined both the second-order Poisson equation and the fourth-order bi-harmonic equation, both of which yield the same solution. We trained networks with varying widths m, ranging from 10 2 to 10 6 , for each combination of p and k using GD optimization with a learning rate of 10 -8 . Figure 1 illustrates the training losses at the initial stage on a logarithmic scale, supporting our theoretical findings that a larger width is needed for higher values of p to ensure convergence. Moreover, we can observe that narrower networks tend to converge more readily when solving lower-order PDEs (Poisson) compared to higher-order PDEs (bi-harmonic). This observation aligns with Theorem 4.3 that higher-order PDEs necessitate larger network widths for guaranteed convergence.

Validation on the effect of p To verify the influence of the power p of the ReLU activation function, we test PINNs with varying p values between 3 and 10 . Since the training process became highly unstable as p increases, we consider second-order heat equation (509) [14] to gain a more precise investigation of the effect of p . The results are summarized in Figure 2 (a). We can see that the convergence of loss is enhanced as p decreases, which supports our theoretical finding.

Comparison between PINNs and VS-PINNs To validate the order reduction effect of VS-PINNs, we conducted an experiment comparing PINNs with VS-PINNs on the second-order heat equation. Each model was run five times with different random seeds, and Figure 2 depicts the training loss for both PINNs and VS-PINNs along with their variance. The results show that the training loss for VS-PINNs converges more effectively than that of PINNs. This indicates that VS-PINNs, which optimize a loss function incorporating lower-order derivatives using networks with smaller p , facilitate convergence of GD, consistent with the theoretical findings in Section 4. Furthermore, we performed a similar experiment on the convection-diffusion equation (511) in the Appendix E and obtained results that were consistent with those observed for the heat equation.

Effect of splitting level For higher-order PDEs, there are several ways to transform a given PDE into a lower-order system through variable splitting. To investigate this effect, we conducted experiments on the fourth-order elastic beam equation (510) [52] with two cases: (i) ϕ 0 ≈ u, ϕ 1 ≈ u t , ϕ 2 ≈ u xx

(a) Effect of

p

(b) Effect of variable splitting

Train losses for1-D heat equation

Heat equation

Beam equation

0.10

PINNS

10.0

PINNS.

VS-PINNs

VS-PINNs||= 2

0.08

1.0

VS-PINNsI| =1

0.1

0.1

Train l

0.06

p=5

p=4

Train

Train

0.01

p=6

0.001

0.04

p=7

0.01

0.0001

p=8

0.02

p=9

1e-05

p=10

0.001

0.00

0

50

100

Epoch (x10)

150

200

250

300

500

Epoch (x100)

3500

4000

500

1000

Epoch (x100)

3000

3500

4000

with | ξ | = 2 and p = 3 and (ii) the finest splitting of ϕ 0 ≈ u, ϕ 1 ≈ ∇ u, ϕ 2 ≈ u xx , ϕ 3 ≈ u xxx with | ξ | = 1 and p = 2 . In order to train PINNs for a fourth-order PDE, p should be at least five, but training such PINNs with GD does not proceed properly, as illustrated in Figure 1. Consequently, the experiments were conducted using the Adam optimizer. In contrast to the underperforming PINNs, VS-PINNs are effectively trained even with GD, as illustrated in Figure 6 of the Appendix E. We run each model five times with different random seeds, and Figure 2 (b) depicts the training loss of PINN and two VS-PINNs with variance. The results show that the model with a lower PDE order k and a smaller power p of the activation exhibits a more pronounced reduction in the loss function, in accordance with our theoretical findings. Furthermore, it can be observed that the variance of the training loss is significantly smaller for the models with smaller values of k and p . This indicates that the learning process is much more stable for a smaller k and p . We also conduct numerical studies on the fourth-order bi-harmonic equation (508). However, the results exhibit a similar trend to that observed in the beam equation and are therefore presented in Appendix E.

6 Conclusion

In this paper, we proved that the gradient flow of PINNs converges to a global minimum and provides sufficient width for this convergence. It extends the results in [25] to general PDEs and activation functions and provides even tighter conditions on the width size. The main theorem demonstrates that the PDE order or dimension exponentially increases the width requirement, theoretically indicating that PINNs are challenging to optimize for high-order or high-dimensional PDEs. We also substantiate that the PDE order amplifies the adverse effects of dimensionality, which explains why PINNs are more susceptible to dimensionality than other deep learning losses without differentiation. Furthermore, We showed that the variable splitting strategy improves convergence by reducing the differential order included in the training loss function.

It is acknowledged that we only provided sufficient conditions for convergence. To fully comprehend the role of these factors in optimizing PINNs, it is also necessary to establish the necessary conditions linking PDE order, dimension, and convergence. Given that the primary goal of PINNs is to approximate the solution of PDEs, it could also be a limitation that all discussions were limited to empirical losses with fixed collocation points. It would therefore be a worthwhile future direction to analyze the conditions under which the expected loss converges when training collocation points are randomly sampled per epoch. Extending our theoretical framework to analyze the impact of the variable splitting strategy on the generalization error of PINNs, as suggested in [64], would also be an interesting and important research direction. Moreover, as our analysis was confined to continuous time flows, a comprehensive understanding of gradient descent would necessitate the analysis of discretized flows, since GF and GD have different dynamics [50]. We expect that our theory could be adapted to GD dynamics by using Theorem 3.3 of [50], which treats GD as GF with a counter term, but we leave it for future work. In a practical context, the convergence of PINNs for adaptive optimizers, such as Adam [35] or L-BFGS [43], and other activation functions, including hyperbolic tangent, remains an open question.

References

Ahmed Abdeljawad and Philipp Grohs. Integral representations of shallow neural network with rectified power unit activation function. Neural Networks , 155:536-550, 2022.

Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, Jose Santamaría, Ahmed Shihab Albahri, Bashar Sami Nayyef Al-dabbagh, Mohammed A Fadhel, Mohamed Manoufali, Jinglan Zhang, Ali H Al-Timemy, et al. A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. Journal of Big Data , 10(1):46, 2023.

Abdon Atangana and Ernestine Alabaraoye. Solving a system of fractional partial differential equations arising in the model of hiv infection of cd4+ cells and attractor one-dimensional Keller-Segel equations. Advances in Difference Equations , 2013:1-14, 2013.

Roberto Ballarini. The Da Vinci-Euler-Bernoulli Beam theory? Mechanical Engineering Magazine Online , 2003.

Guy Barles and Halil Mete Soner. Option pricing with transaction costs and a nonlinear black-scholes equation. Finance and Stochastics , 2:369-397, 1998.

Shamsulhaq Basir. Investigating and mitigating failure modes in physics-informed neural networks (PINNs). Communications in Computational Physics , 33(5):1240-1269, 2023.

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research , 18(153):1-43, 2018.

Feliks Aleksandrovich Berezin and Mikhail Shubin. The Schrödinger Equation , volume 66. Springer Science & Business Media, 2012.

Jiang Bo-Nan and CL Chang. Least-squares finite elements for the stokes problem. Computer Methods in Applied Mechanics and Engineering , 78(3):297-311, 1990.

Laxman Bokati, Vladik Kreinovich, Joseph Baca, and Natasha Rovelli. Why rectified power (repu) activation functions are efficient in deep learning: A theoretical explanation. In Uncertainty, Constraints, and Decision Making , pages 7-13. Springer, 2023.

Kurt Izak M. Cabanilla, Rhudaina Z. Mohammad, and Jose Ernie C. Lope. Neural networks with relu powers need less depth. Neural Networks , 172:106073, 2024.

John W Cahn and John E Hilliard. Free energy of a nonuniform system. i. interfacial free energy. The Journal of chemical physics , 28(2):258-267, 1958.

Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica , 37(12):1727-1738, 2021.

John Rozier Cannon. The one-dimensional heat equation . Number 23. Cambridge University Press, 1984.

Charles K Chui, Xin Li, and Hrushikesh Narhar Mhaskar. Neural networks for localized approximation. mathematics of computation , 63(208):607-623, 1994.

Bernardo Cockburn and Chi-Wang Shu. The local discontinuous galerkin method for timedependent convection-diffusion systems. SIAM journal on numerical analysis , 35(6):2440-2463, 1998.

Peter Constantin. Euler equations, Navier-Stokes equations and turbulence. In Mathematical Foundation of Turbulent Viscous Flows: Lectures given at the CIME Summer School held in Martina Franca, Italy, SEptember 1-5, 2003 , pages 1-43. Springer, 2005.

Tim De Ryck, Florent Bonnet, Siddhartha Mishra, and Emmanuel de Bézenac. An operator preconditioning perspective on training in physics-informed machine learning. arXiv preprint arXiv:2310.05801 , 2023.

Tim De Ryck and Siddhartha Mishra. Error analysis for physics-informed neural networks (pinns) approximating kolmogorov pdes. Advances in Computational Mathematics , 48(6):79, 2022.

Lokenath Debnath and Lokenath Debnath. Nonlinear partial differential equations for scientists and engineers . Springer, 2005.

MWMGDissanayake and Nhan Phan-Thien. Neural-network-based approximations for solving partial differential equations. communications in Numerical Methods in Engineering , 10(3):195201, 1994.

E Kirk Donald et al. Optimal control theory: an introduction. Mineola, NY: Dover Publications, Inc , 1970.

Nathan Doumèche, Gérard Biau, and Claire Boyer. Convergence and error analysis of pinns. arXiv preprint arXiv:2305.01240 , 2023.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Yihang Gao, Yiqi Gu, and Michael Ng. Gradient descent finds the global optima of two-layer physics-informed neural networks. In International Conference on Machine Learning , pages 10676-10707. PMLR, 2023.

Roland Glowinski, Hao Liu, Shingyu Leung, and Jianliang Qian. A finite element/operatorsplitting method for the numerical solution of the two dimensional elliptic monge-ampère equation. Journal of Scientific Computing , 79:1-47, 2019.

John B Greer, Andrea L Bertozzi, and Guillermo Sapiro. Fourth order partial differential equations on general geometries. Journal of Computational Physics , 216(1):216-246, 2006.

Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physicsinformed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering , 379:113741, 2021.

QiZhi He, David Barajas-Solano, Guzel Tartakovsky, and Alexandre M Tartakovsky. Physicsinformed neural networks for multiphysics data assimilation with application to subsurface transport. Advances in Water Resources , 141:103610, 2020.

Masanobu Horie and Naoto Mitsume. Physics-embedded neural networks: Graph neural pde solvers with mixed boundary conditions. Advances in Neural Information Processing Systems , 35:23218-23229, 2022.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks , 3(5):551-560, 1990.

K Hosseini, K Sadri, M Mirzazadeh, YM Chu, A Ahmadian, BA Pansera, and S Salahshour. A high-order nonlinear schrödinger equation with the weak non-local nonlinearity and its optical solitons. Results in Physics , 23:104035, 2021.

Zheyuan Hu, Zekun Shi, George Em Karniadakis, and Kenji Kawaguchi. Hutchinson trace estimation for high-dimensional and high-order physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering , 424:116883, 2024.

Deqing Jiang, Justin Sirignano, and Samuel N Cohen. Global convergence of deep galerkin and pinns methods for solving partial differential equations. arXiv preprint arXiv:2305.06000 , 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering , 358:112623, 2020.

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks , 9(5):9871000, 1998.

Hyuk Lee and In Seok Kang. Neural algorithm for solving differential equations. Journal of Computational Physics , 91(1):110-131, 1990.

Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research , 20(40):1-47, 2019.

Xin Li. Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer. Neurocomputing , 12(4):327-343, 1996.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming , 45(1):503-528, 1989.

Minliang Liu, Liang Liang, and Wei Sun. A generic physics-informed neural network-based constitutive model for soft biological tissues. Computer methods in applied mechanics and engineering , 372:113402, 2020.

Yu Liu and Wentao Ma. Gradient auxiliary physics-informed neural network for nonlinear biharmonic equation. Engineering Analysis with Boundary Elements , 157:272-282, 2023.

El Mehdi Lotfi, Mehdi Maziane, Khalid Hattaf, and Noura Yousfi. Partial differential equations of an epidemic model with spatial diffusion. International Journal of Partial Differential Equations , 2014:1-6, 2014.

Tao Luo and Haizhao Yang. Two-layer neural networks for partial differential equations: Optimization and generalization theory. arXiv preprint arXiv:2006.15733 , 2020.

Revanth Mattey and Susanta Ghosh. A physics informed neural network for time-dependent nonlinear and higher order partial differential equations. arXiv preprint arXiv:2106.07606 , 2021.

Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physicsinformed neural networks for approximating a class of inverse problems for pdes. IMA Journal of Numerical Analysis , 42(2):981-1022, 2022.

Taiki Miyagawa. Toward equation of motion for deep neural networks: Continuous-time gradient descent and discretization error analysis. Advances in Neural Information Processing Systems , 35:37778-37791, 2022.

Eike H Müller and Robert Scheichl. Massively parallel solvers for elliptic partial differential equations in numerical weather and climate prediction. Quarterly Journal of the Royal Meteorological Society , 140(685):2608-2624, 2014.

Frithiof I Niordson. On the optimal design of a vibrating beam. Quarterly of Applied Mathematics , 23(1):47-53, 1965.

Bernt Oksendal. Stochastic differential equations: an introduction with applications . Springer Science & Business Media, 2013.

Yesom Park, Taekyung Lee, Jooyoung Hahn, and Myungjoo Kang. p -Poisson surface reconstruction in curl-free flow from point clouds. Advances in Neural Information Processing Systems , 2023.

Yesom Park, Chang Hoon Song, Jooyoung Hahn, and Myungjoo Kang. ReSDF: Redistancing implicit surfaces using neural networks. Journal of Computational Physics , 502, 2024.

Yesom Park, Changhoon Song, and Myungjoo Kang. Beyond derivative pathology of pinns: Variable splitting strategy with convergence analysis. arXiv preprint arXiv:2409.20383 , 2024.

Shige Peng. A generalized dynamic programming principle and hamilton-jacobi-bellman equation. Stochastics: An International Journal of Probability and Stochastic Processes , 38(2):119-134, 1992.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: Adeep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics , 375:1339-1364, 2018.

Roger Temam. Navier-Stokes equations: theory and numerical analysis , volume 343. American Mathematical Soc., 2001.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055-A3081, 2021.

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, 2022.

Jue Yan and Chi-Wang Shu. Local discontinuous galerkin methods for partial differential equations with higher order derivatives. Journal of Scientific Computing , 17:27-47, 2002.

Marius Zeinhofer, Rami Masri, and Kent-André Mardal. A unified framework for the error analysis of physics-informed neural networks. arXiv preprint arXiv:2311.00529 , 2023.

Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using physics-informed neural networks. Science advances , 8(7):eabk0644, 2022.

A Mathematical Notations

We introduce the symbols and mathematical notations that are frequently used in this paper.

For a positive integer m , the set { 1 , · · · , m } is abbreviated as [ m ] . The set of multi-indexes whose size is m or is at most m is referred by

In this paper, we use two-layer MLPs. For given k ∈ N and a partition 0 = ξ 0 < ξ 1 < · · · < ξ L < ξ L +1 = k , we define ϕ ℓ : R d → R | I ξ ℓ | of width m as

where weights in the first and second layer are

and y = [ x ⊤ 1 2 ] ⊤ ∈ R d +1 . Note that the output dimension of ϕ ℓ is R | I ξ ℓ | because each component of ϕ ℓ ( x ; w ℓ , v ℓ ) represents partial derivative ∂ ξ ℓ ∂ x α u , for each α ∈ I ξ ℓ . Hence, even though ( v ℓ ) r is a flattened vector, it is more convenient to use α ∈ I ξ ℓ as an index of component of ( v ℓ ) r . For example, if d = 2 , ξ 1 = 1 , and α = (0 , 1) ∈ I 1 , ( v 1 ) r,α = ( v 1 ) r, (0 , 1) refers a weight between the

r -th hidden node and output that represents ∂ ∂y u ( x, y ) . Similarly, we use multi-index itself as an index that is related to I ξ ℓ , I ∆ ξ ℓ , or J ∆ ξ ℓ .

Moreover, as each ( ϕ ℓ ) α is differentiated by some ∂ ∆ ξ ℓ ∂ x β for α ∈ I ξ ℓ -1 and β ∈ I ∆ ξ ℓ , index we use pair of multi-index ( α, β ) to refer a component of ∂ ∆ ξ ℓ ∂ x β ( ϕ ℓ ) r,α . Sometimes, this indexing by multi-index is used in conjunction with normal indexes, like ( µ ℓ ) ( α,β ) ,i for some i ∈ [ n o ] .

We denote the collection of weights of all ϕ ℓ 's by w = [ w ⊤ 0 , · · · w ⊤ L ] ⊤ and v = [ v ⊤ 0 , · · · v ⊤ L ] ⊤ . Similarly, µ is the collection of all µ ℓ 's, µ = [ µ ⊤ 0 , · · · , µ ⊤ L ] . With regularization parameters ν, ν ℓ > 0 , residuals s i , ( µ ℓ ) i , and h j of variable splitting for each training sample

and

define the loss L V S PINN for the system,

we define

a

({

|

ˆ

a

ℓ,α,β

(

x

i

)

|

:

0

α

ℓ

∈

I

ξ

ℓ

L, i

, β

∈

∈

[

J

n

o

∆

ξ

]

ℓ

α

j

∈

∈

J

[

1

n

,

b

]

B Calculations

The proof of the main theorem includes intricate calculations. To keep the proof clear, we separate some tedious computations that are used frequently. As this section is a reference for the proofs, we use some notation or symbols without any mention if they are defined in the other part of the paper.

Sizes of index set I m and J m . For any ℓ ∈ [ L ] ,

We bound | I ξ ℓ | , | ∆ ξ ℓ | and J ∆ ξ ℓ by

,

}

∪

{

|

˜

a

α

(˜

x

j

)

|

:

})

,

(28)

Partial derivative of ˆ s , µ , and h . Note that ∂ β ∂ x β ( ( w ℓ ) ⊤ r y i ) p + in (34), ∂ β ∂ x β ( ( w ℓ -1 ) ⊤ r y i ) p + in (35), ( ( w ℓ ) ⊤ r y i ) p + in (36), and ∂ α ∂ x α ( ( w 0 ) ⊤ r ˜ y j ) p + in (37) are polynomial of ( w ℓ ) r , ( w ℓ -1 ) r , ( w ℓ ) r , and ( w 0 ) r of degree p , respectively.

For i ∈ [ n o ] , ℓ = 0 , 1 , . . . , L , r ∈ [ m ] , and α ∈ I ξ ℓ ,

Similarly, for ℓ ∈ [ L ] , r ∈ [ m ] , α 1 ∈ I ξ ℓ -1 , α 2 ∈ I ξ ℓ -1 , β ∈ I ∆ ξ ℓ , and i ∈ [ n o ] , we have

For ℓ ∈ [ L ] , r ∈ [ m ] , α 1 ∈ I ξ ℓ , α 2 ∈ I ξ ℓ -1 , β ∈ I ∆ ξ ℓ , and i ∈ [ n o ] ,

For r ∈ [ m ] and j ∈ [ n b ] ,

̸

and for r ∈ [ m ] and j ∈ [ n b ] ,

Similarly, for ℓ ∈ [ L ] , r ∈ [ m ] , and i ∈ [ n o ] , we have the following inequalities: for α ∈ I ξ ℓ ,

̸

for ℓ ′ = ℓ -1 , ℓ , and

where

As s i,r is a p -th degree polynomial of ( w ℓ ) r , we have

for all i ∈ [ n o ] and r ∈ [ m ] .

for α 1 ∈ I ξ ℓ , α 2 ∈ I ξ ℓ -1 , and β ∈ I ∆ ξ ℓ ,

Hoeffding's inequalities for s i,r , ( µ ℓ ) r, ( α,β ) ,i , and h j,r . For each i ∈ [ n o ] and r ∈ [ m ] , let

for all i ∈ [ n o ] and r ∈ [ m ] .

Using E [ ( v ℓ ) r,α ] = 0 , we attain E [ s i,r ( w (0) , v (0))] = 0 , and Hoeffding's inequality gives

for all ε > 0 .

Similarly, for each ℓ ∈ [ L ] , α ∈ I ξ ℓ -1 , β ∈ I ∆ ξ ℓ , and i ∈ [ n o ] ,

where

Then, we attain that

and

E [ ( v ℓ ) r,α ] = 0 implies E [ ( µ ℓ ) r, ( α,β ) ,i ] = 0 and for each i ∈ [ n o ] , Hoeffding's inequality with the sum over r ∈ [ m ] of variables

with

gives

For each j ∈ [ n b ] , let us denote

where

It is clear that h jr is expressed as a p -th order polynomial of ( w 0 ) r and we can deduce that

for all j ∈ [ n b ] and r ∈ [ m ] .

Similar to the case of s i,r and ( µ ℓ ) r, ( α,β ) ,i , Hoeffding's inequality gives

Components of Q ℓ 1 ,ℓ 2 Each component of Q ℓ 1 ,ℓ 2 is polynomial of order p as follows:

The number of possible nonzero elements in Q ℓ 1 ,ℓ 2 and ̂ G v . There are possibly n 2 o , ∣ ∣ I ξ ℓ -1 ∣ ∣ | I ∆ ξ ℓ | n 2 o , and n o n b nonzero elements in Q 0 , 0 , Q 0 ,ℓ , and Q 0 ,L +1 , respectively. For each ( α 1 , β 1 ) ∈ I ξ ℓ -1 × I ∆ ξ ℓ , there are at most | I ∆ ξ ℓ | pairs of ( α 2 , β 2 ) ∈ I ξ ℓ -1 × I ∆ ξ ℓ such that α 1 + β 1 = α 2 + β 2 and at most | I ∆ ξ ℓ | pairs of ( α 2 , β 2 ) ∈ I ξ ℓ -1 × I ∆ ξ ℓ such that α 1 = α 2 . Thus, there are at most 2 | I ∆ ξ ℓ | pairs of ( α 2 , β 2 ) ∈ I ξ ℓ -1 × I ∆ ξ ℓ with α 1 + β 1 = α 2 + β 2 or α 1 = α 2 , and Q ℓ,ℓ has possibly (∣ ∣ I ξ ℓ -1 ∣ ∣ | I ∆ ξ ℓ | n o ) (2 | I ∆ ξ ℓ | n o ) = 2 ∣ ∣ I ξ ℓ -1 ∣ ∣ · | I ∆ ξ ℓ | 2 n 2 o nonzero elements. Similarly, each ( α 1 , β 1 ) ∈ I ξ ℓ -1 × I ∆ ξ ℓ uniquely determines α 2 = α 1 + β 1 ∈ I ξ ℓ , there are ∣ ∣ I ∆ ξ ℓ +1 ∣ ∣ pairs of ( α 2 , β 2 ) ∈ I ξ ℓ × I ∆ ξ ℓ +1 with α 2 = α 1 + β 1 . Hence, Q ℓ,ℓ +1 has ∣ ∣ I ξ ℓ -1 ∣ ∣ · | I ∆ ξ ℓ | · ∣ ∣ I ∆ ξ ℓ +1 ∣ ∣ n 2 o nonzero elements. Q 1 ,L +1 has | I ∆ ξ 1 | n o n b nonzero elements. The number of nonzero elements in Q L +1 ,L +1 is n 2 b . After all, the number of nonzero elements in ̂ G v is

at most N 1 = 7 ( d + k d )( d + | ξ | d ) 2 n 2 o +4 ( d + | ξ | d ) n o n b + n 2 b , because

Norms of partial derivative of Q . For each i 1 , i 2 ∈ [ n o ] , r ∈ [ m ] , and ℓ ∈ { 0 } ∪ [ L ] , we have

For each ℓ ∈ [ L ] , α ∈ I ξ ℓ -1 , β ∈ I ∆ ξ ℓ , and i 1 , i 2 ∈ [ n o ] ,

and

̸

For each i ∈ [ n o ] and j ∈ [ n b ] ,

̸

For each ℓ ∈ [ L ] , α 1 , α 2 ∈ I ξ ℓ -1 , β 1 , β 2 ∈ I ∆ ξ ℓ , and i 1 , i 2 ∈ [ n o ] ,

and

̸

Similarly, for each ℓ ∈ { 1 , · · · , L -1 } , α 1 ∈ I ξ ℓ -1 , α 2 ∈ I ξ ℓ , β 1 ∈ I ∆ ξ ℓ , β 2 ∈ I ∆ ξ ℓ +1 and i 1 , i 2 ∈ [ n o ] ,

̸

(194)

For each ℓ ∈ [ L ] , α = 1 , β ∈ I ∆ ξ 1 , i ∈ [ n o ] and j ∈ [ n b ] ,

1

ν

n

o

ν

n

b

∑

r

=1

) ( ∑

α

∈

J

1

̸

For each j 1 , j 2 ∈ [ n b ] , the direct calculations lead to

̸

(

w

)

⊤

r

y

˜

a

(

w

)

⊤

r

˜

y

1

,

(197)

= 1

√

(

∂

∂

β

x

β

(

)

p

+

∂

∂

α

x

α

m

0

i

α

(

0

j

)

p

+

{

ℓ

}

=1

Hoeffding's inequalities for Q ℓ 1 ,ℓ 2 . For each i ∈ [ n o ] and j ∈ [ n b ] , we have

Then, for each i 1 , i 2 ∈ [ n o ] ,

By Hoeffding's inequality with

we have

for any ε > 0 .

Similarly, for each ℓ ∈ [ L ] , α ∈ I ξ ℓ -1 , β ∈ ∆ ξ ℓ , and i 1 , i 2 ∈ [ n o ] ,

By Hoeffding's inequality with

we have

P [∣ ∣ ∣ ( Q 0 ,ℓ ) i 1 , (( α,β ) ,i 2 ) -E [ ( Q 0 ,ℓ ) i 1 , (( α,β ) ,i 2 ) ]∣ ∣ ∣ > ε ] < 2 exp ( -mn 2 o ε 2 2 ν ℓ ( d + | ξ | d ) 2 ( p | ξ | +1 ) 2 a 2 max p 2 | ξ | R 4 p ) , (228) for any ε > 0 .

For each i ∈ [ n o ] and j ∈ [ n b ] ,

By Hoeffding's inequality with

we have

for any ε > 0 .

For each ℓ ∈ [ L ] , α 1 , α 2 ∈ I ξ ℓ -1 , β 1 , β 2 ∈ J ∆ ξ ℓ , and i 1 , i 2 ∈ [ n o ] ,

ℓ

⊤

r

i

1

+

⊤

r

i

2

+

α

1

+

β

1

=

α

2

+

β

2

∣

∣

∣

By Hoeffding's inequality with

ℓ

{

}

+

(

w

)

y

(

w

)

y

1

(235)

we have

for any ε > 0 .

By Hoeffding's inequality with

we have

P [∣ ∣ ∣ ( Q ℓ,ℓ +1 ) (( α 1 ,β 1 ) ,i 1 ) , (( α 2 ,β 2 ) ,i 2 ) -E [ ( Q ℓ,ℓ +1 ) (( α 1 ,β 1 ) ,i 1 ) , (( α 2 ,β 2 ) ,i 2 ) ]∣ ∣ ∣ > ε ] < 2 exp ( -mn 2 o ε 2 2 ν ℓ ν ℓ +1 p 2 | ξ | R 4 p ) , (245) for any ε > 0 .

For each α ∈ I ξ ℓ -1 , β ∈ J ∆ ξ ℓ , i ∈ [ n o ] , and j ∈ [ n b ] ,

By Hoeffding's inequality with

we have

for any ε > 0 .

For j 1 , j 2 ∈ [ n b ] ,

By Hoeffding's inequality with

we have

C Proofs

This section proves the main theorem with some lemmas and propositions. Similar to the [25], we first prove that the Gram matrix ̂ G ∞ v is independent of m and strictly positive definite, in Proposition C.3. Second, as ̂ G v ( w (0) , v (0)) is the sample mean, Proposition C.4 shows that ̂ G v ( w (0) , v (0)) is close to its expected value ̂ G ∞ v with a high probability, if m is large enough. This implies that ̂ G v ( w (0) , v (0)) is strictly positive definite with high probability. Then, we show that the initial loss is bounded with a high probability if m is large. This is because v (0) has zero mean, which hinders the output of m variables to diverge as m increases. Finally, we prove that the Gram matrix ̂ G v ( w ( t ) , v ( t )) remains within a small neighborhood of the initial point, where it maintains its strict positive definiteness. The gradient flow converges within the neighborhood, as the smallest eigenvalue of ̂ G v is distinct from zero, resulting in a rapid reduction of the loss.

Lemma C.1. Let d, n ∈ N and y 1 , . . . , y n ∈ R d be vectors such that y i / / y j if and only if i = j . Then, for each i ∈ [ n ] , there exists w i such that w ⊤ i y j = 0 if and only if i = j .

̸

Proof. Let y ⊥ i = { w ∈ R d : w ⊤ y i = 0 } . Since y i / / y j for i = j , y i ∩ y j is nowhere dense in y ⊥ i . Hence, finite union of ∪ j = i y ⊥ j is nowhere dense in y ⊥ i , and in particular, there exists w i ∈ y ⊥ i such that w i ̸∈ ∪ j = i y ⊥ j .

̸

Lemma C.2. Let n, d ∈ N , and ψ : R d → R be a homogeneous polynomial of degree n . For any i ∈ N and nonzero y = ( y 1 , . . . , y d ) ∈ R d , if a function ψ ( w ) ( w ⊤ y ) i + of w = ( w 1 , . . . , w d ) ∈ R d is ( n + i ) -times continuously differentiable at some w ∗ ∈ R d with ( w ∗ ) ⊤ y = 0 , then ψ ≡ 0 .

̸

Proof. Without loss of the generality, we may assume the y 1 = 0 . Write ψ ( w ) = ∑ α ∈ I n a α w α for coefficients a α ∈ R and define φ ( w ) = ψ ( w ) ( w ⊤ y ) i + . For any α ∈ I n , the assumption on differentiability implies that ∂ i ∂w i 1 ∂ n ∂ w α φ is continuous. Indeed, we have φ ( w ) = 0 for w ⊤ y ≤ 0 , and

for w ⊤ y > 0 . The continuity induces a α = 0 and thus ψ ≡ 0 for all w ∈ R d .

Proposition C.3. ̂ G ∞ v = E w , v [ ̂ G v ( w , v ) ] is strictly positive definite and independent of m .

Proof. For i 1 , i 2 ∈ [ n o ] ,

and m random variables

̸

for r ∈ [ m ] , are independent and identically distributed. Therefore, the expected value of each component of Q 0 , 0 is independent of m , as the expected value of the sample mean of independent and identically distributed random variables is independent of the number of samples. Similarly, each component of ̂ G v ( w , v ) is the sample mean of m independent and identically distributed random variables, and its expected value ̂ G ∞ v is independent of m . To show that ̂ G ∞ v is strictly positive definite, assume that there exists z 0 = [ ( z 0 ) 1 · · · ( z 0 ) n o ] ∈ R n o , ˜ z = [˜ z 1 · · · ˜ z n b ] ⊤ ∈ R n b , and

for ℓ ∈ [ L ] , such that z = [ z ⊤ 0 z ⊤ 1 · · · z ⊤ L ˜ z ⊤ ] ⊤ ∈ R n o + ∑ L ℓ =1 ( | I ξ ℓ -1 | · | I ∆ ξ ℓ | n o ) + n b satisfies ̂ G ∞ v z = 0 . Since ̂ G ∞ v z = E w , v [ ˆ D ⊤ v ˆ D v ] z = 0 if and only if E w , v [ ∥ ∥ ∥ ˆ D v z ∥ ∥ ∥ 2 2 ] = 0 , we show that E w , v [ ∥ ∥ ∥ ˆ D v z ∥ ∥ ∥ 2 2 ] = 0 implies z = 0 and thereby 0 is not an eigenvalue of ̂ G ∞ v .

Now assume that there exists z ∈ R d such that ˆ D v z = 0 for almost all w and v . Since ˆ D v is continuous with respect to w and v , we have ˆ D v z = 0 for all w and v . Hence, the function ˆ D v z of w and v , which is identically zero, is smooth with respect to w and v .

In this proof, we denote an index for a component of ˆ D v z by ℓ ∈ [ L ] , r ∈ [ m ] and α ∈ I ξ ℓ , because each row of ˆ D v corresponds to the partial derivatives with respect to ( v ℓ ) r,α for ℓ ∈ [ L ] , r ∈ [ m ] and α ∈ I ξ ℓ . For instance, ( ˆ D v z ) 0 , 1 , 0 is the first component of ( ˆ D v z ) . First, for ℓ = L and any r ∈ [ m ] and α ∈ I ξ L , we have

̸

Fix i ∈ [ n o ] , and by Lemma C.1, there exists ( w ∗ L ) r ∈ R d such that ( w ∗ L ) ⊤ r y i = 0 , ( w ∗ L ) ⊤ r y i ′ = 0 for i = i ′ ∈ [ n o ] and ( w ∗ L ) ⊤ r ˜ y j = 0 for j ∈ [ n b ] . As a function of ( w L ) r , ˆ D v z = 0 implies that

̸

̸

̸

̸

is smooth at ( w L ) r = ( w ∗ L ) r . Note that the coefficient of ( ( w L ) ⊤ r y i ) p -∆ ξ L +1 + in (266) is given by

which is a homogeneous polynomial of w L . Together with the fact that at least one of ˆ a L,α,β ( x i ) is nonzero, Lemma C.2 implies ( z 0 ) i = 0 . Similarly, for any r ∈ [ m ] ,

̸

̸

deduces ˜ z j = 0 with w ∗ 0 , instead of w ∗ L , such that ( w ∗ 0 ) ⊤ ˜ y j = 0 , ( w ∗ 0 ) ⊤ ˜ y j ′ == 0 for j ′ = j , and ( w ∗ 0 ) ⊤ y i = 0 for i ∈ [ n o ] . Now suppose that we have ˜ z = 0 and z ℓ ′ = 0 for ℓ ′ = 0 , 1 , . . . , ℓ -1 for some ℓ ≥ 1 . Then, for any r ∈ [ m ] and α ∈ I ξ ℓ -1 ,

̸

Then the coefficient of ( ( w ℓ ) ⊤ r y i ) p -∆ ξ ℓ is a homogeneous polynomial

Lemma C.2 shows ( z ℓ ) ( α,β ) ,i = 0 for all α, β . The induction concludes z = 0 .

In the sequel, λ 0 = λ min ( ̂ G v ) > 0 is the minimal eigenvalue of ̂ G v . Furthermore, we assume the occurrence of the following event: there exists R > 1 such that ∥ ( w ℓ ) r (0) ∥ 2 < R and ∥ ( v ℓ ) r (0) ∥ 2 < R for all ℓ = 0 , 1 , . . . , L and r ∈ [ m ] . This assumption is employed solely for the purpose of simplifying propositions and their respective proofs. In the main theorem and its proof, we use the following arguments with regard to the conditional probability of the aforementioned event.

Proposition C.4. Let δ > 0 and

be constants. If m is large enough so that

then with the probability of at least 1 -δ over the initialization, we have

Proof. We use Hoeffding's inequality to measure the error between ̂ G v ( w , v ) and ̂ G ∞ v , as Gram matrix ̂ G ∞ v is the expectation of the sample mean. In the Hoeffding's inequalities (220), (228), (233), (241), (245), (251), and (254), we have

Consequently, all inequalities induce that each component of ∣ ∣ ∣ ̂ G v -̂ G ∞ v ∣ ∣ ∣ is greater than ε , with a probability of at most 2 exp ( -mε 2 2 C 2 1 R 4 p ) . Since there exists at most N 1 nonzero elements in ∣ ∣ ∣ ̂ G v -̂ G ∞ v ∣ ∣ ∣ by (145),

holds componentwise, with the probability of at least

Then, we have

Set ε and m to satisfy ε = 1 4 √ N 1 λ 0 and 2 N exp ( -mε 2 2 C 2 1 R 4 p ) < δ . In other words, if

By the above proposition, the initial Gram matrix ̂ G v ( w (0) , v (0)) is likely to be strictly positive definite, and its smallest eigenvalue remains greater than λ 0 2 , at the beginning of the flow. The following lemma implies that at the early stage of the flow, for which w and v are not far from the initial values, the positive definiteness is preserved, and its smallest eigenvalue remains distinct from zero.

Lemma C.5. Suppose ∥ ˜ w ℓ -w ℓ (0) ∥ 2 ≤ R w < R , for

Then

Proof. 1. Case of Q 0 , 0 : By the mean value theorem with (151), we can induce that

for some w ∗ = [ ( w ∗ 0 ) ⊤ 1 · · · ( w ∗ 0 ) ⊤ m · · · ( w ∗ L ) ⊤ 0 · · · ( w ∗ L ) ⊤ m ] ⊤ lies on a line between ˜ w and w (0) and for some v ∗ lies on a line connecting ˜ v and v (0) . Since ∥ ∥ ( w ∗ ℓ ) r ∥ ∥ 2 ≤ ∥ ( ˜ w ℓ ) r -( w ℓ ) r (0) ∥ 2 + ∥ ( w ℓ ) r (0) ∥ 2 ≤ R w + R ≤ 2 R for each ℓ ∈ { 0 } ∪ [ L ] and r ∈ [ m ] ,

Therefore, we can attain that

2. Case of Q 0 ,ℓ : Similar to Q 0 , 0 , the mean value theorem with (159) and (163) gives

Consequently, we have

Case of Q 0 ,L +1 : The mean value theorem with (171) induces

and hence

Case of Q ℓ 1 ,ℓ 2 : From the mean value theorem with (180) and (184), we attain

and further induces

(192) follows that

It deduces that

In accordance with the definition of the loss µ ℓ , it is clear that Q ℓ 1 ,ℓ 2 is identical to the zero matrix if the difference between ℓ 1 and ℓ 2 is greater than 1. Consequently, together with (325) and (333) we have

5. Case of Q ℓ,L +1 : From (201), we have

By the definition of the loss µ ℓ , we have Q ℓ,L +1 is identical to the zero matrix for every ℓ > 1 . Therefore, we have

6. Case of Q L +1 ,L +1 :

From (205), we have

It further follows that

Therefore, combining (302), (311), (313), (339), (347), and (351) concludes that

Proof. Since we have

The preceding lemma indicates that the loss will decrease rapidly in cases where w is not significantly distant from w (0) . Indeed, the subsequent lemma and proposition demonstrate that w ( t ) remains within the designated region for any given value of t > 0 , provided sufficiently large m .

Proposition C.6. Set constants C 2 and C 3 as

For any C > 0 and δ < 2 N 2 exp ( -C 2 3 2 C 2 2 R 2 p ) , if m is large enough so that

then, with the probability of at least 1 -δ over the initialization, we have

Hence, from

with the probability of at least

all the conditions

are satisfied. Then, the square of the initial loss is bounded as

For δ > 0 and C > 0 , if ε and m satisfy the following inequalities,

then for enough small δ such that

we have that with the probability of at least 1 -δ ,

In other words, if

with the probability of at least 1 -δ , we have

Proposition C.7. Suppose the conditions (281) holds and R w are given as in Lemma C.5. For

if m is large enough so that then

for all t > 0 , r ∈ [ m ] and ℓ = 0 , · · · , L .

Proof. We begin the proof by defining

Since ( w ℓ ) r and ( v ℓ ) are continuous, the above set in supremum is nonempty. For t ∈ (0 , T ) ,

where the last inequality comes from Lemma C.5 and the positive semi-definiteness of ̂ G w . This implies that

For t ∈ (0 , T ) and ℓ ∈ { 0 , · · · , L } , from ∥ ( w ℓ ) r (0) ∥ 2 ≤ R and ∥ ( v ℓ ) r (0) ∥ 2 ≤ | I ξ ℓ | and (398), we have

From (44), (57), (52), and (48), we can attain

for all t ∈ (0 , T ) .

Therefore, we can induce that

and

̸

̸

The first term and second term are bounded by

The third term is bounded by

and the fourth term is

Therefore, for t ∈ (0 , T ) and ℓ = { 0 , . . . , L } ,

̸

̸

On the other hand,

and hence

If m is large enough to satisfy (395), we have

Similarly, we obtain

̸

̸

The first term and second term are bounded by

and

a

The third term is bounded by

and the fourth term is

Therefore, for t ∈ (0 , T ) and ℓ = { 0 , . . . , L } ,

√

ν

m

+

d

+

d

p

(2

R

)

∥

s

∥

2

(471)

̸

̸

As a consequence, we attain

d

k

)(

d

|

ξ

)

|

|

ξ

|

p

Since m is large enough to satisfy (395), we have

C.1 Proof of Theorem 4.3

Proof. For R ∈ R , r ∈ [ m ] , and q ∈ [ d ] , each ( w ℓ ) r,q satisfies

If we set δ 1 = ( L +1) md exp ( -1 2 R 2 ) , then δ 1 satisfies

Hence, with probability of at least 1 -δ 1 , we have ∣ ∣ ∣ ( w ℓ ) r,q ∣ ∣ ∣ ≤ R for all ℓ = { 0 , 1 , . . . , L +1 } , r ∈ [ m ] , and q ∈ [ d ] .

Provided ∣ ∣ ∣ ( w ℓ ) r,q ∣ ∣ ∣ ≤ R for all ℓ = { 0 , 1 , . . . , L +1 } , if

then by Proposition C.4, with probability of at least 1 -δ 2 , we have

On the other hand, with the condition ∣ ∣ ∣ ( w ℓ ) r,q ∣ ∣ ∣ ≤ R for all ℓ = { 0 , 1 , . . . , L +1 } , r ∈ [ m ] , and q ∈ [ d ] , Proposition C.6 shows that if

where

where

then, with the probability of at least 1 -δ 3 , we have

Consequently, Proposition C.7 implies

for all t ≥ 0 . Then, we have

from Lemam C.5.

Set δ 1 = δ 2 = δ 3 = 1 3 δ and consider all inequality conditions. First, for δ 1 = 1 3 δ ,

satisfies the condition (483). Then, (484) is satisfied if

Similarly, if m is large enough so that

then (488) holds.

Therefore, there exsits C 5 = C 5 ( a max , ν max , n o , n b , λ 0 ) such that if

then with the probability of at least (1 -δ 1 ) (1 -δ 2 ) (1 -δ 3 ) ≥ 1 -( δ 1 + δ 2 + δ 3 ) = 1 -δ , we have

D Experimental Details

D.1 Problem Formulations

Poisson equation

where f 1 : (0 , π ) 2 → R is defined so that the exact solution is given by

Bi-harmonic equation

where the flux function f 2 : (0 , π ) 2 → R is set to ensure that it has a solution that is identical to that of Poisson equation (507).

Heat equation

Elastic beam equation

The exact solution is given by u ( t, x ) = 2 e -t sin ( x ) .

Convection-diffusion equation

whose exact solution is u ( t, x ) = e -1 4 t sin ( x -t ) .

Initial 40000 epochs

Total 400000 epochs

PINNs(lr=0.01)

PINNs(lr=0.01)

VS-PINNs(lr=0.1)

VS-PINNs(lr=0.1)

0.1

0.1

loss

loss

0.01

Train

Train

0.01

0.001

0.001

0.0001

0.0001

1e-05

50

100

500

4000

Epoch (x100)

Epoch (x100)

D.2 Parameter settings for experiments

Convection-diffusion equation We conducted experiments on a convection-diffusion equation: We train 100,000 epochs of PINNs with p = 3 and VS-PINNs with p = 2 , using the same settings as represented for the heat equation (509). Figures 3 and 4 show that VS-PINNs reach lower loss and achieve more stable convergence for both GD and Adam.

Bi-harmonic equation Consider the bi-harmonic equation

where f ( x, y ) are defined so that the exact solution is given by u ( x, y ) = 1 5 sin 2 ( x ) y 2 ( π -y ) 2 . We set m = 1 , 000 , n o = 10 , 000 , n b = 400 , and the training collocation points are fixed once they are randomly selected. We experiment VS-PINNs with two cases: (i) ϕ 0 ≈ u , ϕ 1 ≈ ( u xx , u yy ) with | ξ | = 2 and p = 3 and (ii) the finest splitting of ϕ 0 ≈ u , ϕ 1 ≈ ∇ u , ϕ 2 ≈ ( u xx , u yy ) , ϕ 3 ≈ ( u xxx , u xxy , u yyy ) with | ξ | = 1 and p = 2 . Regularization parameters are ν 1 = ν 2 = ν 3 = 1 for derivative matching loss µ and ν = 10 for boundary loss h . Figure 5 depicts the training loss of PINN and two VS-PINNs.

E Additional Experimental Results

Computational efficiency of VS-PINNs Table 3 measures the GPU memory, running time, and the number of model parameters corresponding to experiments on elastic beam and bi-harmonic equations. Because VS-PINNs need as many networks as auxiliary variables, finer VS-PINN requires

Initial 10000 epochs

Total 100000 epochs

1

PINNs

PINNs

VS-PINNS

0.1

VS-PINNS

0.1

0.01

loss

loss

0.01

0.001

Train

Train

0.001

0.0001

1e-05

0.0001

1e-06

1e-05

1e-07

0

9

0

400

600

800

1000

Epoch (x100)

Epoch (x100)

Initial40,000epochs

Total400,000epochs

PINNs

PINNs

100.0

VS-PINNs||=2

100.0

VS-PINNs||=2

VS-PINNs || = 1

VS-PINNs || = 1

10.0

10.0

Train loss

loss

1.0

Train

1.0

0.1

0.1

0.01

0.01

0.001

50

350

500

3000

3500

4000

Epochs(x100)

Epochs(x100)

more parameters to be trained. However, reducing the differentiation order in the Loss significantly reduces memory consumption, despite the additional loss terms.

Error with true solution As the ultimate objective of PINNs is to approximate the solution to the PDE, we also quantified the discrepancy between the exact solution of the predicted solutions of the three models we tested. We tested five times for each PDE and splitting setup from different random seeds, and the averages of the Mean Squared Error (MSE) with the exact solution are reported in Table 4. We can observe that the results do not exhibit the same tendency as depicted in Figure 2. Since there is a discrepancy between optimization ability and the generalization error, it is not within the scope of this paper, and we believe that this is an important area for future research.

GD results for fourth-order PDEs In order to address fourth-order PDEs (elastic beam and bi-harmonic equations), PINNs should employ p = 5 of the power of the ReLU p activation. When optimized with GD, this configuration results in a loss value of NaN, indicating that PINNs fail to train. Consequently, we present the results of our experiments with the Adam optimizer in Section 5. However, in contrast to PINNs, which exhibit difficulty in training, variable splitting demonstrates superior performance in terms of loss reduction for GD, as illustrated in Figure 6. This is attributed to the fact that variable splitting employs a lower derivative order and lower power p .

F Broader Impacts

We analyzed the behavior of PINNs when applied to general k th-order PDEs and provided theoretical understanding of the reasons why PINNs encounter difficulties in optimizing when dealing with high-order or high-dimensional problems. This paper has societal impact as it bridges artificial intelligence and physics, and hence has a wide range of potential applications. It could contribute to the advancement in scientific understanding, and foster scientific progress and innovation, benefiting

PDE

Elastic Beam

Bi-harmonic

PINNs

3.0598E-08

3.9319E-05

VS-PINNs

|

ξ

|

= 2

2.2868E-08

5.3055E-05

VS-PINNs

|

ξ

|

= 1

4.3826E-07

5.4418E-03

researchers in physics, mathematics, and artificial intelligence fields. By providing variable splitting which can address the convergence pathologies of PINNs, the paper suggests potential applications in industrial and technological fields, such as engineering, medicine, materials science, and environmental modeling, for complex system modeling, prediction, and optimization. Furthermore, as many problems have arisen in sciences and engineering tied with complex PDE systems, we expect that our work has the potential to be applicable in the enormous area such as climate forecasting, epidemics, molecular simulations, micro-mechanics, and modeling turbulent flows. Additionally, PDEs can be utilized in the development of military equipment. As with all numerical algorithms, however, it is not a work of developing a technique to go to warfare; rather, it is a tool for scientific inquiry. We encourage users of our model to focus on the positive impact of this work.

Initial 40,000 epochs

Total400,000 epochs

80.0

80.0

lr=0.01

lr=0.001

lr=0.0001

60.0

60.0

loss

loss

lr=0.01

Train

40.0

lr=0.001

Train

40.0

lr=0.0001

20.0

20.0

0.0

0.0

50

000t

Epochs(x100)

Epochs(x100)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Building upon the work of Gao et al. [22], the authors extend the analysis of the gradient flow (GF) of PINNs (two-layer MLP is assumed) to general $k^\\mathrm{th}$ order partial differential equations (PDEs) and the $p^\\mathrm{th}$ power of Rectified Linear Unit (ReLU) activation function with general $p$.\n- The authors achieve tighter bounds than those obtained by Gao et al [22].\n- The width of the network necessary for the convergence of GF increases exponentially with the power $p$ of $\\mathrm{ReLU}^p$. \n- The optimal power $p$ is determined by the order $k$ of the governing PDE; $p = k +1$.\n- The GF convergence of PINNs deteriorates with increasing dimensions.\n- To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy, a conventional order-reduction approach.\n\n- An order reduction through variable splitting is proposed based on the theoretical findings above.\n\nOverall, I enjoyed reading the manuscript; however, it would benefit from some revisions.\n\nSTRENGTHS:\n- The topic is relevant and very interesting. The theoretical relationship between PDE's orders and PINN's convergence has been elusive but is a practically important. A challenge comes from the fact that PINN's objective contains the derivatives of PINN's outputs w.r.t. its inputs (Gao et al. [22] is one of pioneering works in this sense).\n\n- Theoretical contributions include:\n  - Building upon the work of Gao et al., the authors extend the analysis of the GF of PINNs, composed of two-layer MLPs, to general $k^\\mathrm{th}$ order PDEs and the $p^\\mathrm{th}$ power of ReLU activation function with general $p$. The authors achieve tighter bounds than those obtained by Gao et al.\n  - The width of the network necessary for the convergence of GF increases exponentially with the power $p$ of $\\mathrm{ReLU}^p$. \n  - The optimal power $p$ is determined by the order $k$ of the governing PDE; $p = k +1$.\n  - The GF convergence of PINNs deteriorates with increasing dimensions.\n  - To address these challenges, we mathematically demonstrate the efficacy of a variable splitting strategy, a conventional order-reduction approach.\n\n- This paper is easy to follow.\n\n- The code is submitted for reproduction, which is important because studies on PINNs are often hard to reproduce due to their strong dependence on random seeds. However, the submitted code does not work, unfortunately (see Weaknesses below).\n\nWEAKNESSES:\n### Major comments\n\n- The most critical concern is that experiments were conducted using Adam, not GD; thus, Section 5 does not validate the theoretical results, which are based on GF. \n  - While training PINNs with GD is challenging (as noted in line 320), I strongly recommend including experimental results using GD, which would significantly enhance the paper's quality. Although some preceding papers have used Adam or other irrelevant optimizers to validate their theoretical results built upon GF, such experiments are, in my opinion, obviously ill-advised.\n  - A continuous-time limit of Adam is given in [Malladi et al., 2022 (https://arxiv.org/abs/2205.10287)], which is, of course, different from GF.\n\n- While GF is easy to handle in theory, GF's underlying assumptions are too restrictive in practice. GF is be a good approximation only when (1) PINNs are trained without random collocation points or are trained on random collocation points without resampling, (2) full-batch training is assumed, and (3) sufficiently small learning rates are used. There three conditions are too restrictive in view of current PINN experiments in the literature. Convergence analyses of DNNs are often criticized for their gaps between theory and practice. I greatly appreciate if these concerns are resolved.\n  - Condition (1) is mentioned in line 358. I would like to ask a question about this point, together with Condition (2), in Questions below.\n  - Condition (3) is mentioned in line 361. It is shown in [Miyagawa, 2022 (https://openreview.net/forum?id=qq84D17BPu)] that theoretical predictions of gradient flow deviate from experimental results as $t \\rightarrow \\infty$ even when learning rates are impractically small. This work would be very helpful and significantly extend the present work, adapting it to practical experimental settings. Please see Questions below.\n  \n\n### Code\nOverall, I would recommend that the authors follow the official code submission guidelines and templates (https://github.com/paperswithcode/releasing-research-code).\n\n- The 'utils' module is missing in the submitted codes, and they do not work.\n\n- Adding requirements.txt would be recommended.\n\n\n### Minor comment\n\n- $\\mathrm{ReLU}^p$ is called the rectified power unit (RePU) in [Bokati et al. (https://scholarworks.utep.edu/cgi/viewcontent.cgi?article=2717&context=cs_techrep)]. Please see also the references therein. I would recommend citing them in the submitted paper.\n\n\n\n\n### Typos\n- Line 154: Impact of PDE order on convergence of PINNs -> Impact of PDE Order on Convergence of PINNs\n- Line 203: $\\delta << 1$ -> $\\delta \\ll 1$\n- Line 372: keller-segel -> Keller-Segel\n- Line 373: da vinci-euler-bernoulli -> da Vinch-Euler-Bernoulli\n- Line 414: pinns -> PINNs\n- Line 462: i -> I\n- Line 507: Dgm -> DGM\n\nQUESTIONS:\n- Can the theoretical analysis given in this paper generalize to other activation functions, e.g., leaky $\\mathrm{ReLU}^p$, softplus, and GeLU, sine, etc.?\n\n- Can the theoretical analysis given in this paper generalize beyond two-layer networks?\n\n- How can we extend this work to non-gradient-flow-based optimization problems, which are more prevalent and practical, e.g., random collocation points with resampling (i.e., SGD-like training)?\n\n- To what extent theoretical predictions deviate from its corresponding experimental results when SGD-like training is used?\n\n- As mentioned above, it is shown that theoretical predictions of gradient flow deviate from experimental results as $t \\rightarrow \\infty$ even when learning rates are impractically small as shown in [Miyagawa, 2022 (https://openreview.net/forum?id=qq84D17BPu)]. How can we alleviate this issue, or is this possibly not a problem in the present paper?\n\n- (Footnote 3 on page 5) Is there any other evidence or related work that smoother activation functions require larger networks for convergence? The opposite, i.e., \"smoother activation functions help convergence and *reduce* the required size of width,\" would also sound valid in the context of general DNN training. I would greatly appreciate if the authors discuss this point further.\n  - Inequality (10) and (16) seem to be *sufficient* conditions for convergence and are not necessary (please correct me if I missed something). Therefore, the intuitive explanation in footnote 3 on page 5 would be valid only in limited cases.\n\nLIMITATIONS:\nYes (described in Section 6).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary:**\n\nThe paper discusses the convergence of the gradient flow of a PINN loss function for arbitrary-order linear PDEs. The analyzed neural networks are shallow MLPs with a power-of-ReLU activation function. The results established are of the NTK convergence type in the overparameterized regime and take the following form: Assume the width mm of the network is larger than a critical number $C$, i.e.,\n$$\n\tm > C\n$$ \nthen, with high probability with respect to the initialization, the PINN loss will converge to zero when the neural network parameters follow the gradient flow of the typical PINN loss function.\n\nThe authors analyze the dependence of the constant $C$ on the PDE order, the dimension of the computational domain, and the power of the activation function. It is proven that a high PDE order, high dimension, and a large power of the employed ReLU activation deteriorate the convergence, i.e., require a larger constant $C$ to guarantee the convergence of the gradient flow. This leads the authors to propose first-order reformulations which, as proven in the manuscript and illustrated with numerical examples, mitigate the aforementioned problems. The main novelty of the preprint lies in the quantification of the constant $C$.\n\n**General Impression:**\nThe paper is carefully written. Although the proofs are long and technical and deferred to the Appendix, the main part is an enjoyable and understandable read. However, I have a major issue with how the findings are communicated; see below in the paragraph on the weaknesses of this reply.\n\nSTRENGTHS:\n1. Well written paper, understandable to read for a broad audience albeit technical proofs.\n2. Addresses a challenging problem (training of PINNs) and tries to provide theoretical guidance on how to improve the training process.\n\nWEAKNESSES:\n1. There is no sharpness of the lower bound $m > C$. Thus, the conclusions the authors draw from the lower bound could potentially be wrong. This is briefly touched upon in the conclusion section of the paper, but I strongly recommend changing the presentation; see the discussion below.\n2. Achieving zero training loss is one step in a comprehensive mathematical analysis of PINNs. Other aspects, such as the discussion of the approximation error, the quadrature error, and the coercivity properties of the PINN loss function, complete the picture. I believe the authors can improve their preprint by contextualizing it better and providing pointers to the literature. Below, I have attached a list of references and briefly discussed their relevance to the overall theory.\n\n**Expansion on 1.**\n\nDrawing conclusions from a lower bound of the form $m>C$ is dangerous as for non-sharp estimates (like the present one) one can not be sure whether dependencies of $C$ on the data (PDE order, dimensionality etc) are artefacts of the proof or are truly required to obtain a lower bound. An exception would be if the lower bound is sufficiently tight, such that it is of practical value. This is not the case here: Setting $d=1$, $k=1$, $p=2$ and neglecting the log term (setting it to unity) yields\n$$\nm \\geq 2^{14} \\cdot 2^{11} \\cdot 2^{12} = 2^{37} \\approx. 10^{11}. \n$$\n\nTherefore, I strongly advise against presenting the results in a way that suggests a connection between the theoretical results (how $C$ depends on $p, k$ and $d$) and the observed optimization struggle of PINNs in the literature. Portraying results in this way leads to folklore in the community which is not backed by rigorous mathematics. At the very least, I advise the authors to clarify this early on in the introduction and before drawing conclusions from Theorem 3.2. I acknowledge that the authors discuss the limitations in the Conclusions section, but this is not where it belongs.\n\n**Expansion on 2.**\n\nThe presented results should be better embedded in the existing literature. As a matter of fact, this will strengthen the authors points, especially with respect to the PDE reformulation in first-order systems. I recommend the following articles:\n\n- [1] The article discusses error analysis/coercivity estimates for PINNs with sharp coercivity estimates. See for instance equation (3.10) in [1]. In this equation $L$ is the loss and $\\eta$ quantifies the mismatch between discrete loss (analyzed in the authors preprint) and population loss, which behaves according to Monte-Carlo integration and can be analyzed by Rademacher complexity arguments, as for example illustrated in [2]. Furthermore, the validity of equation 3.10 in [1] relies on coercivity properties of the PDE at hand and is illustrated for a number of linear PDEs in [1]. With the aforementioned articles, the errors analysis of PINNs is more comprehensive and I strongly believe readers should be pointed in this direction when reading the authors preprint.\n\n- The benefit of first-order reformulations of PINN type losses is well-known in the literature, also for different reasons than the ones the authors point out. For example [3] reports better results for jumping material coefficients and shows that strong formulations will not work at all in this case. Another aspect is treated in [1]. It is shown that first order reformulations lead to improved convergence rates, compare the error estimates for  equation to the ones of Poisson equation. To achieve $H^{1/2}$ convergence for Poisson, one needs approximation in $H^2$, for $H^{1/2}$ convergence for Darcy (which can be viewed as a first order reformulation of Poisson) one needs approximation in $H^1$ for the pressure and $H(\\operatorname{div})$ for the velocity, which leads to improved rates.\n\nOverall, I think the paper merits publication if the weaknesses are addressed and I am willing to raise my score in this case.\n\nReferences:\n\n[1] https://arxiv.org/abs/2311.00529\n\n[2] https://arxiv.org/pdf/2308.16429\n\n[3] https://www.sciencedirect.com/science/article/abs/pii/S0021999120304812\n\nQUESTIONS:\nSee weaknesses discussion.\n\nLIMITATIONS:\nSee weaknesses discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
This paper presents a theoretical analysis of the relation between PDE order and the convergence of PINNs. A tighter bound is obtained than the previous work. Inspired by the importance of reducing PDE order, the authors propose the VS-PINN, which employs the variate splitting strategy. Both theoretical analysis and empirical results are included to prove the effectiveness of the proposed method.

STRENGTHS:
-	Overall, I think this paper is theoretically solid and interesting.

-	The idea of using variable splitting is reasonable.

WEAKNESSES:
1.	Practical efficiencies are expected for comparison, such as GPU memory, running time and model parameters. Although VS-PINN does not require the calculation of high-order derivatives, the newly added regularization term in Eq. 13 will also bring extra computation costs.

2.	All the conclusions in Section 3 can be directly obtained from the previous paper [22]. I do not think the tighter bound brings better theoretical insights in terms of understanding “How does PDE order affect the convergence of PINNs?”

3.	How about model performance in typical PDEs, such as Convection or Reaction?

QUESTIONS:
-	I suggest the authors provide a clear description of the proof for Theorem 3.2 and show how they obtain a tighter bound.

-	I think the equation in line 247 has a typo, which is $l\in[0, L+1]$.

-	How to choose $L$ in practice?

LIMITATIONS:
I appreciate that they have discussed the limitations. There are some more powerful and advanced backbones, such as PINNsFormer [1]. More experiments on them will further demonstrate the effectiveness of the proposed method although they are hard to analyze in a theoretical aspect.

[1] PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks, ICLR 2024
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical understanding about the behavior of PINN when dealing with high-order or high-dimesional PDEs. Variable splitting is then proposed to decomposes the high-order PDE into a system of lower-order PDEs and facilitate the convergence of PINN.\n\nSTRENGTHS:\n1. This paper extends Gao et al.'s work to higher order PDEs and ReLU activation functions and demonstrates that the gradient flow of PINNs converges to the global minimum with a high probability when the width of the network is sufficiently large.\n2. Using the derived bound, this paper comprehensively analyzes the impact of PDE order and dimensionality on the behaviour of PINNs.\n3. The numerical results further demostrates the theoretical results and validates the positive effect of utilizing variable splitting.\n\nWEAKNESSES:\n1. The numerical results pertaining to the parameter  p  indicate a contradiction. Specifically, the convergence of the PINN is observed to be faster with  p = 4  compared to  p = 3 , which stands in contrast to the assertion that “the convergence of the loss is enhanced as  p  decreases.” Additionally, there are typographical errors in Figure 1(a), where the legends should display  p  rather than  k.\n2. The derived bound demonstrates the substantial impact of both the PDE order and the activation power on the network’s width, indicating that as  p  increases, the width  m  must also increase exponentially to ensure convergence. However, the numerical experiments conducted in the paper do not vary the network width  m  to empirically verify this theoretical relationship. Instead, they maintain a fixed width across different experiments, focusing primarily on the effects of varying the power  p  of the ReLU activation function and the impact of variable splitting. While these experiments do validate the theoretical findings related to  p  and variable splitting, they fall short of demonstrating how changes in network width influence convergence behavior.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}