Title: Dual Cone Gradient Descent for Training Physics-Informed Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: physics-informed neural networks;multi-objective optimization;scientific machine learning;gradient descent
Abstract: Physics-informed neural networks (PINNs) have emerged as a prominent approach for solving partial differential equations (PDEs) by minimizing a combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs often fail to generate reasonable solutions, and such pathological behaviors remain difficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, we propose a novel optimization framework, *Dual Cone Gradient Descent* (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in a non-convex setting. On a variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms of various evaluation metrics. In particular, DCGD achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD can be further improved by combining it with popular strategies for PINNs, including learning rate annealing and the Neural Tangent Kernel (NTK).

Dual Cone Gradient Descent for Training Physics-Informed Neural Networks

Abstract

Physics-informed neural networks (PINNs) have emerged as a prominent approach for solving partial differential equations (PDEs) by minimizing a combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs often fail to generate reasonable solutions, and such pathological behaviors remain difficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, we propose a novel framework for multi-objective optimization, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in a non-convex setting. On a variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms of various evaluation metrics. In particular, DCGD achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD can be further improved by combining it with popular strategies for PINNs, including learning rate annealing and the Neural Tangent Kernel (NTK). Codes are available at https://github.com/youngsikhwang/Dual-Cone-Gradient-Descent .

1 Introduction

Physics-informed Neural Networks (PINNs) proposed in Raissi et al. [1] have created a new paradigm in deep learning for solving forward and inverse problems involving partial differential equations (PDEs). The key idea of PINNs is to integrate physical constraints, governed by PDEs, into the loss function of neural networks. This is in turn equivalent to finding optimal parameters for the neural network by minimizing a loss function that combines boundary loss and PDE residual loss. Thanks to their strong approximation ability and mesh-free advantage, PINNs have achieved great success in a wide range of applications [2-8].

Building upon this success, the applications of PINNs have been extended to solve other functional equations, including integro-differential equations [9], fractional PDEs [10], and stochastic PDEs [11]. Moreover, numerous variants of PINNs have been developed to enhance their computational efficiency and accuracy via domain decomposition methods [12, 13], advanced neural network architectures [14-18], modified loss functions [19-21], different sampling strategies [22-24], and

probabilistic PINNs [25, 26]. Recent studies have also explored optimizing PINNs by leveraging function space geometry, providing an alternative perspective to enhance accuracy and computational efficiency [27, 28].

Despite these achievements, several studies have reported that PINNs often fail to learn correct solutions for given problems ranging from highly complex to relatively simple PDEs [29-31]. Due to the unclear nature of pathologies in the training of PINNs, it has become a critical research topic to explain and mitigate these phenomena. For example, [32, 33] observed that PINNs tends to get stuck at trivial solutions while violating given PDE constraints over collocation points. The imbalance between PDE residual loss and boundary loss was explored in Wang et al. [30], and a spectral bias of PINNs was studied in Wang et al. [31]. Yao et al. [34] discussed the gap between the loss function and the actual performance. Even with the insights from the aforementioned studies, a comprehensive understanding of PINN's failure modes remains largely unexplored in various scenarios.

In this paper, we explore these challenges from a novel perspective of multi-objective optimization. We first provide a geometric analysis showing that PINNs can be adversely trained when the gradients of each loss function exhibit a significant imbalance in their magnitudes, coupled with a negative inner product value. Based on this finding, we characterize a dual cone region where both PDE residual loss and boundary loss can be decreased simultaneously without the adverse training phenomenon. We then propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), for training PINNs which updates the gradient direction to be contained in the dual cone region at each iteration. Furthermore, we study the convergence properties of DCGD in a non-convex setting. In particular, we find that DCGD can converge to a Pareto-stationary point. We validate the superior empirical performance and universal applicability of DCGD through extensive experiments.

2 Preliminaries

Notation. The Euclidean scalar product is denoted by ⟨· , ·⟩ , with ∥ · ∥ standing for the Euclidean norm (where the dimension of the space may vary depending on the context). For a subspace W of a vector space V , its orthogonal complement W ⊥ is defined as

For a vector v ∈ V , the projection of v on a nontrivial subspace W is denoted by v ∥ W . Unless otherwise specified, V represents R d throughout the paper.

Related Works. Among various research directions in PINNs, we focus on reviewing optimization strategies for PINNs. These can be broadly categorized into three main approaches: adaptive loss balancing, gradient manipulation, and Multi-Task Learning (MTL). As an example of adaptive loss balancing algorithms, Wang et al. [30] proposed a learning rate annealing (LRA) algorithm that balances the loss terms by utilizing gradient statistics. Wang et al. [31] utilized the eigenvalues of the Neural Tangent Kernel (NTK) to address the disparity in convergence rates among different losses of PINNs. For gradient manipulation algorithms, the Dynamic Pulling Method (DPM) was proposed in [35] to prioritize the reduction of the PDE residual loss. In [36], the authors used the PCGrad algorithm, proposed in [37], for training PINNs to address multi-task learning challenges. Li et al. [38] developed an adaptive gradient descent algorithm (AGDA) that resolves the conflict by projecting boundary condition loss gradient to the normal plane of the PDE residual loss gradient. Yao et al. [34] recently developed MultiAdam, a scale-invariant optimizer, to mitigate the domain scaling effect in PINNs. Another important line of gradient manipulation involves Multi-Task Learning (MTL) algorithms, which optimize a single model to perform multiple tasks simultaneously [39, 40, 37, 41-44]. We will discuss that several MTL algorithms can be unified within the proposed DCGD framework.

Physics-Informed Neural Networks. Let Ω ⊆ R D be a domain and ∂ Ω be the boundary of Ω . We consider the following nonlinear PDEs:

where N and B denote a nonlinear differential operator and a boundary condition operator, respectively. We approximate u ( x ) by a deep neural network u ( x ; θ ) parameterized by θ . To train the

neural network, the framework of PINNs minimizes the total loss function L ( θ ) , which is a weighted sum of PDE residual loss L r ( θ ) and boundary condition loss L b ( θ ) , defined by:

where ω r , ω b ≥ 0 are weights of each loss term, { x i r } N r i =1 denotes a set of collocation points that are randomly sampled in Ω , and { x i b } N b i =1 the boundary sample points. Here, we set ω r = ω b = 1 throughout the paper. We note that the training of PINNs falls into the category of multi-objective learning due to its structure of the loss function L ( θ ) in Eq. (2.2).

3 Empirical Observations and Issues in Training PINNs

This section investigates issues that are frequently observed during the training of PINNs in the context of multi-objective learning. The parameter for the PINN solution u ( x ; θ ) is typically estimated by minimizing the total loss function L ( θ ) with a (stochastic) gradient descent method 2 :

where ∇L ( θ ) is the gradient of the total loss function L ( θ ) with respect to θ . However, a careless adoption of standard gradient descent methods may lead to an incorrect solution, as reducing the total loss does not necessarily

0

500

1000

1500

2000

2500

3000

3500

4000

Epoch

0.0

0.1

0.2

0.3

0.4

Loss

b

r

if

K

*

if

K

*

imply a decrease in both the PDE residual loss and boundary loss. This phenomenon is clearly illustrated in Figure 1, which displays the curves of the total loss, PDE residual loss, and boundary loss over epochs for solving the viscous Burger's equation. Notably, while the total loss consistently decreases throughout the training, the PDE loss adversely increases.

Conflicting and dominating gradients in PINNs. This issue is highly related with discrepancies in the direction and magnitude between two gradients of the PDE residual and boundary loss. Specifically, we define two gradients to be conflicting at the t -th iteration if they have a negative inner product value, i.e., π 2 < ϕ t ≤ π where ϕ t is the angle between ∇L r ( θ t ) and ∇L b ( θ t ) . When there are conflicting gradients, parameter updates to minimize one loss function might increase the other, leading to an inefficient learning process such as oscillating between optimizing for the two loss functions and resulting in degraded solution quality [19]. Another problem arises when one gradient is much larger than the other, i.e., ∥∇L r ( θ t ) ∥ ≪ ∥∇L b ( θ t ) ∥ or ∥∇L r ( θ t ) ∥ ≫ ∥∇L b ( θ t ) ∥ . The significant differences 3 in the magnitudes of gradients in PINNs might create a situation where the optimization algorithm primarily minimizes one loss function while neglecting the other. This often results in slow convergence and overshooting, as the smaller gradient, though neglected, may be more crucial in finding a better solution. To mitigate the imbalance in the gradients, loss balancing approaches to rescale the weights of each loss term have been proposed [30, 31].

To examine these challenges in training PINNs, we record cosine value of the angle between ∇L r and ∇L b , and the ratios of their magnitudes while training a PINN for the Helmholtz equation. Figure 2(a) shows that conflicting gradients are observed in about half of the total iterations. Moreover, we observe that the magnitude of the gradient of the PDE residual is several tens to hundreds of times larger than that of the boundary loss (See Figure 2(b)). That is, conflicting and dominating gradients are prevalent issues in the training of PINNs.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

cos(

)

0.0

0.2

0.4

0.6

0.8

Density

(a) histogram of

cos(

ϕ

)

0

50

100

150

200

250

||

r

||/||

b

||

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Density

(b) histogram of

R

4 Methodology

In this section, we provide a geometric analysis to identify a dual cone region where both the PDE residual loss and the boundary loss can decrease simultaneously. Subsequently, we introduce a general framework for DCGD algorithms, ensuring that the updated gradient falls within this region. We then propose three specifications of DCGD algorithms: projection, average, and center. All proofs for main results in this section can be found in Appendix A.

4.1 Dual Cone Region

The concept of a dual cone plays a pivotal role in our DCGD algorithm. Formally, a dual cone is defined as a set of vectors that have nonnegative inner product values with a given cone.

Definition 4.1. (Dual cone) Let K be a cone of R d . Then, the set

is called the dual cone of K .

For each iteration t , consider a cone denoted by K t , which is generated by rays of two gradients, ∇L r ( θ t ) and ∇L b ( θ t ) :

In the context of PINNs, the dual cone of K t , denoted by K ∗ t , represents the set of gradient vectors where each vector is neither conflicting with the gradient of the PDE loss nor with the gradient of the boundary loss, i.e., for u ∈ K ∗ t , ⟨ u, ∇L r ( θ t ) ⟩ ≥ 0 and ⟨ u, ∇L b ( θ t ) ⟩ ≥ 0 .

In other words, when the total gradient ∇L ( θ t ) is in K ∗ t (as depicted by the region of the red line in Figure 1), the standard gradient descent taking the direction ∇L ( θ t ) will decrease both the PDE and boundary losses for a suitable step size. On the other hand, if ∇L ( θ t ) / ∈ K ∗ t (the region indicated by the blue line in Figure 1), one of the two losses will adversely increase even with sufficiently small step sizes.

This indicates that the training process of PINNs can significantly vary depending on whether the total gradient belongs to the dual cone region. The following theorem establishes the necessary and sufficient conditions under which the total gradient falls within the dual cone region in terms of the angle and relative magnitude between the gradients of the PDE residual and boundary loss.

Theorem 4.2. Suppose that ∇L r ( θ t ) and ∇L b ( θ t ) are given at each iteration t . Let ϕ t be the angle between ∇L r ( θ t ) and ∇L b ( θ t ) , and R = ∥∇L r ( θ t ) ∥ ∥∇L b ( θ t ) ∥ be their relative magnitude. Then, ∇L ( θ t ) ∈ K ∗ t if and only if

( i ) ⟨∇L ( θ ) , ∇L ( θ ) ⟩ ≥ 0 , or

Theorem 4.2 provides a clear criterion for when conflicting and dominating gradients lead to adverse training in PINNs. For instance, the condition (ii) in Theorem 4.2 implies that the larger ϕ t (the more conflicting they are), even a slight difference in their magnitudes can result in adverse training. In particular, Theorem 4.2 quantifies the extent of problematic relative magnitude between the two

K*

Gt

VL(0t)

VCr(0t)

gradients, thereby clarifying the concept of dominating gradients, which has not been previously defined in the literature.

Thus, our strategy aims to devise an algorithm that chooses the updated gradient within the dual cone region at each gradient descent step. For notational simplicity, we write ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b to represent ∇L ( θ t ) ∥ ( ∇L r ( θ t )) ⊥ and ∇L ( θ t ) ∥ ( ∇L b ( θ t )) ⊥ , respectively. In particular, we are interested in a simple and explicit subspace G t , defined as the set of conic combinations of ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b :

for two reasons. Firstly, all vectors in G t are easily computable due to the explicit expression of G t , whereas the dual cone K ∗ is implicitly defined. Secondly, G t contains two important components of K ∗ t , which are the projections of ∇L ( θ t ) onto ∇L r ( θ t ) ⊥ and ∇L b ( θ t ) ⊥ by its construction. The next proposition shows that G t always belongs to the dual cone region as illustrated in Figure 3.

Proposition 4.3. Suppose that ∇L r ( θ t ) and ∇L b ( θ t ) are given at each iteration t . Consider G t , the set of conic combinations of ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b , defined in Eq. (4.1). Then, G t ⊆ K ∗ t .

Consequently, the DCGD algorithm defines the updated gradient denoted by g dual t within G t at each iteration t . A general framework for DCGD is presented in Algo 1.

Algorithm 1 Dual Cone Gradient Descent (base)

end for

4.2 Convergence Analysis

To discuss the convergence properties of DCGD, we introduce the concept of Pareto optimality (adapted to the PINN setting), which is a key in multi-objective optimization [46, 40].

Definition 4.4. (Pareto optimal and stationary) A point θ ∈ R d is said to be Pareto-optimal if there does not exist θ ′ ∈ R d such that

In addition, a point θ ∈ R d is said to be Pareto-stationary if there exists α 1 , α 2 such that

Intuitively, a Pareto-stationary point implies there is no feasible descent direction that would decrease all loss functions simultaneously. For example, consider a point θ t at which the cosine of the angle ϕ t between ∇L r ( θ t ) and ∇L b ( θ t ) is -1 , i.e., cos( ϕ t ) = -1 . Such a point is Pareto-stationary.

The following theorem guarantees the convergence of the DCGD algorithm proposed in Algo 1 under some regularities in a non-convex setting. Assume L ( θ ∗ ) := inf θ ∈ R d L ( θ ) > -∞ .

Theorem 4.5. Assume that both loss functions, L b ( · ) and L r ( · ) , are differentiable and the total gradient ∇L ( · ) is L -Lipschitz continuous with L > 0 . If g dual t satisfies the following two conditions:

(ii) There exists M > 0 such that ∥ g dual t ∥ ≥ M ∥∇L ( θ t ) ∥ ,

then, for λ ≤ 1 2 L , DCGD in Algo. 1 converges to a Pareto-stationary point, or converges as

Theorem 4.5 states that DCGD converges to either a Pareto-stationary point, characterized by ϕ t such that cos( ϕ t ) = -1 , or a stationary point at a rate of O (1 / √ T ) in the nonconvex setting. Unlike singleobjective (nonconvex) optimization where the goal is to pursue a stationary point, in multi-objective optimization, it is ideal to find a Pareto-stationary point that balances all loss functions. Thus, DCGD offers significant theoretical and empirical advantages over popular optimization algorithms like SGD and ADAM, which are only guaranteed to converge to a stationary point. The convergence of DCGD to a Pareto-stationary point is empirically verified in Section 4.4.

4.3 Dual Cone Gradient Descent: Projection, Average, and Center

(a) DCGD (Projection)

(b) DCGD (Average)

(c) DCGD (Center)

Gt

Gt

Gt

=V

gdual

VtCC

VL(0t)

(0)△

VL(0t)

VC(0t)

(0)△

(0)*△

2

2

(0)*7△

Different variants of DCGD can be designed by properly choosing the updated gradient g dual t in G t satisfying the conditions (i), (ii) of Theorem 4.5. We present three specific algorithms: projection, average, and center.

The first algorithm, named DCGD (Projection), uses the projection of the total gradient ∇L ( θ t ) onto G t when ∇L ( θ t ) / ∈ K ∗ t , which is the closest vector within G t to ∇L ( θ t ) . Specifically, the DCGD (Projection) algorithm specifies g dual t as follows: (i) ∇L ( θ t ) if ∇L ( θ t ) ∈ K ∗ t , (ii) ∇ t L ∥∇L ⊥ r ( c 1 = 1 , c 2 = 0 ) if ∇L ( θ t ) / ∈ K ∗ t and ⟨∇L ( θ t ) , ∇L r ( θ t ) ⟩ < 0 , (iii) ∇ t L ∥∇L ⊥ b ( c 1 = 0 , c 2 = 1 ) if ∇L ( θ t ) / ∈ K ∗ t and ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ < 0 . See also Eq. (E.1) and Algo. 2.

DCGD (Average) algorithm takes the average of ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b when the total gradient is outside K ∗ t , i.e., c 1 = c 2 = 1 2 if ∇L ( θ t ) / ∈ K ∗ t . See Eq. (E.2) and Algo. 3.

We note that both DCGD (Projection) and DCGD (Average) use ∇L ( θ t ) as g dual t without any manipulation when ∇L ( θ t ) ∈ K ∗ t . Moreover, they require determining if the total gradient is contained in the dual cone at each iteration, which may incur additional computational costs. On the other hand, g dual t of DCGD (Center) is given by

which is geometrically interpreted as the projection of ∇L ( θ t ) onto the angle bisector g c t of ∇L r ( θ t ) and ∇L b ( θ t ) . The following proposition shows that g dual t of DCGD (Center) resides within G t

Proposition 4.6. Consider the updated gradient g dual t of DCGD (Center) defined in Eq. (4.3). Then, g dual t ∈ G t .

The visualization of these three algorithms can be found in Figure 4 and their pseudocodes are provided in Appendix E. Moreover, the proposed DCGD algorithms satisfy the conditions (i) and (ii) of Theorem 4.5. Consequently, the following Corollary summarizes the convergence of the proposed DCGD algorithms.

Corollary 4.7. We impose the same assumptions as in Theorem 4.5. Then, DCGD (Projection), DCGD (Average), and DCGD (Center) converge to either a Pareto-stationary point or a stationary point.

In addition to the theoretical result in Corollary 4.7, Appendix D.1 provides an ablation study on the empirical performance of three specific algorithms for solving benchmark PDEs.

0.75

0.50

0.25

0.00

0.25

0.50

0.75

cos(

t

)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Density

(a) ADAM

0.0

0.2

0.4

0.6

0.8

1.0

cos(

t

)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Density

(b) DCGD (Projection)

0.0

0.2

0.4

0.6

0.8

cos(

t

)

0

5

10

15

20

25

Density

(c) DCGD (Average)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

cos(

t

)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Density

(d) DCGD (Center)

4.4 Benefits of the DCGD framework

This subsection discusses benefits of DCGD through illustrative examples. We first investigate how the proposed DCGD algorithms resolve the conflicting gradient issue discussed in Section 3. Given each algorithm, at each iteration t , we define φ r t as the angle between the updated vector and ∇L r ( θ t ) , and φ b t as the angle between the updated vector and ∇L b ( θ t ) . Also, let φ max t = max { φ r t , φ b t } . We highlight that both φ r t and φ b t are less than π/ 2 under DCGD algorithms, as they ensure that the updated vectors always belong to the dual cone. Figure 5 plots the distributions of cos( φ max t ) for four different optimization algorithms: ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center) during the training of PINNs for solving the Helmholtz equation. It shows that three DCGD algorithms completely eliminate conflicting gradients in contrast to ADAM. Moreover, we observe that the distributions of cos( φ max t ) for DCGD (Projection) and DCGD (Average) are highly skewed toward zero, which implies that one of the two losses is unlikely to significantly improve. On the contrary, DCGD (Center) has a bell-shaped distribution with a mean of about 0.719, indicating that the two gradients are more aligned. This leads to a continuous reduction in both losses in a harmonious manner. Consistent with this observation, DCGD (Center) consistently outperforms DCGD (Projection) and DCGD (Average) in our experiments. Please refer to the ablation study D.1 for further comparisons.

We empirically demonstrate that DCGD can converge to a Pareto-stationary point. Consider a (slightly modified) toy example shown in [37, 41], which has two objective functions; see Appendix C.2 for more details. We solve the problem with 1,600 uniformly sampled initial points using ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center). Then, we mark with a red dot the point at which the algorithm fails to reach a Paretostationary point. Figure 6 shows that while ADAM does not reach a Pareto-stationary point across many areas, all DCGD algorithms achieve convergence to Paretostationary points throughout the entire space.

Several MTL algorithms, such as PCGrad [37], MGDA [40], CAGrad [41], Aligned-MTL [44], and

10

5

0

5

10

1

10

5

0

5

10

2

(a) ADAM

10

5

0

5

10

1

10

5

0

5

10

2

(b) DCGD

Nash-MTL [43] have been developed based on different and independent approaches. In contrast, the proposed DCGD framework provides a principled solution to the problem of conflicting gradients by directly characterizing the dual cone. As a result, our framework unifies many of these MTL

algorithms as special cases, offering significant contributions not only to PINNs but also to the MTL domain. Proofs for the unification of MTL algorithms within the DCGD framework can be found in Appendix B.

5 Numerical Experiment

This section demonstrates the superiority of DCGD through three distinct perspectives. In Section 5.1, we compare the performance of DCGD on five benchmark equations with that of a range of methods, including ADAM [45], Learning Rate Annealing (LRA) [30], Neural Tangent Kernel (NTK) [31], PCGrad [37], MGDA [40], CAGrad [41], Aligned-MTL [44], MultiAdam [34], and DPM [35]. Section 5.2 shows that DCGD can provide more accurate solutions for failure modes of PINNs and complex PDEs where vanilla PINNs fail. In Section 5.3, we explore the compatibility of DCGD with existing loss balancing schemes such as LRA and NTK.

To compare the effectiveness of DCGD with other optimization algorithms, we measure the accuracy of the PINN solution trained by each optimizer using the relative L 2 -error. Then, we run each experiment across 10 independent trials and report the mean, standard deviation, max, and min of the best accuracy.

5.1 Comparison on benchmark equations

We solve three popular benchmark equations (the Helmholtz equation, the viscous Burgers' equation, and the Klein-Gordon equation) and two high-dimensional PDEs (5D-Heat equation and 3D-Helmholtz equation) using vanilla PINNs with different optimization techniques. For DCGD, we employ an adaptive gradient version of the DCGD (Center) algorithm, the DCGD (Center) combined with ADAM (see Algo 5) by default for all experiments, provided in Appendix D.1. For other methods, we perform careful hyperparameter tuning based on the recommendations in their papers. The PDE equations and detailed experimental setting are provided in Appendix C.4. However, we do not report the performance of DPM because it is not only highly sensitive to hyperparameters but also exhibit poor performance, consistently observed in [47].

Table 1 displays the mean and standard deviation of the relative L 2 errors for each optimization algorithm applied to the three PDE equations. The error plots of approximated PINN solutions and other statistics of relative L 2 errors are summarized in Appendix C.4. In the result tables, we highlight the best and the second-best methods. While the second best methods vary across experiments, the proposed method consistently outperforms other algorithms achieving the lowest L 2 errors. This result underscores the robustness and adaptability of our method for solving various PDEs.

5.2 Failure mode of PINNs and Complex P(I)DEs

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

t

2

0

2

1(

t

0) = 0.55,

2(

t

0) = 0.81

IC

Reference:

1,

1

Reference:

2,

2

PINN:

1,

1

PINN:

2,

2

(a) SGD

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

t

2

0

2

1(

t

0) = 2.72,

2(

t

0) = 2.55

IC

Reference:

1,

1

Reference:

2,

2

PINN:

1,

1

PINN:

2,

2

(b) ADAM

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

t

2

0

2

1(

t

0) = 2.62,

2(

t

0) = 2.62

IC

Reference:

1,

1

Reference:

2,

2

PINN:

1,

1

PINN:

2,

2

(c) DCGD

We explore more challenging problems, including failure modes of PINNs and complex PDEs, where vanilla PINNs fail to approximate solutions, and highlight the universal applicability of DCGD. We refer to Appendix C.5 for detailed experimental settings.

First, we revisit the problem of a double pendulum in Steger et al. [32], which is highly sensitive to initial conditions. The goal is to solve the trajectory of

{ ( θ 1 ( t ) , θ 2 ( t )) } t ≥ t 0 , governed by the nonlinear differential equation as discussed in Eq. (C.1). The reference solution and its first-order derivative are represented by the blue solid and dotted lines, respectively, in Figure 7. We train PINNs with SGD and ADAM to solve the double pendulum problem, where their solutions are depicted by the red solid and dotted lines in Figure 7a and Figure 7b, respectively. The PINN solutions trained with SGD and ADAM fail to accurately approximate the reference solution. In contrast, the reference solution is successfully recovered by our DCGD algorithm (see Figure 7c). Second, we present the performance of DCGD for two challenging PDEs: the chaotic Kuramoto-Sivashinsky (KS) equation and the convection equation. For the chaotic KS equation, we combine DCGD with the causal training scheme of [22], the current state-of-the art result. For the convection equation, DCGD is applied to PINNsFormer of [15]. As shown in Table 2, DCGD achieves the lowest relative L 2 errors for the complex PDEs compared to the existing optimally tuned strategies, demonstrating its effectiveness in overcoming failure modes of PINNs. Third, the universal applicability of DCGD is not limited to specific architectures, sampling techniques, and training schemes. For example, A-PINN, designed for solving integral equations and integro-differential equations, achieves state-of-the art results in nonlinear V olterra IDEs [9]. DCGD significantly improves the performance of A-PINN for solving Volterra IDEs, as shown in Table 2. Moreover, Table 4 shows that the performance of SPINN can be highly improved by applying DCGD for solving multi-dimensional PDEs.

5.3 Compatibility of DCGD with existing methods

The proposed DCGD framework can be easily combined with existing PINN training strategies, including loss balancing methods. To illustrate this advantage, we have designed DCGD algorithms that integrate with LRA and NTK, named DCGD (Center) + LRA and DCGD (Center) + NTK, respectively. Please refer to Algo. 6 for the detailed implementation.

We apply DCGD (Center) + LRA and DCGD (Center) + NTK to the same experiments described in Section 5.1. Tables 1 and 3 demonstrate that the performance of DCGD algorithms can be further enhanced across all the experiments in terms of the mean, maximum, and minimum of relative L 2 errors by integrating existing ideas from the literature.

6 Conclusion and Discussion

In this work, we provided a clear criterion for when PINNs might be adversely trained, in terms of the angle and relative magnitude ratio of the gradients of the PDE residual and boundary loss, through a geometric analysis. Based on this theoretical insight, we characterized a dual cone region where both losses can decrease simultaneously without gradient pathologies. We then proposed a general

framework for DCGD, which ensures that the updated gradient falls within the dual cone region, and provided a convergence analysis. Within this general framework, we introduced three specific DCGD algorithms and conduct extensive empirical experiments. Our experimental results demonstrate that the proposed DCGD algorithms outperform other optimization algorithms. In particular, DCGD is efficient in solving challenging problems such as failure modes of PINNs and complex PDEs compared to the current state-of-the art approaches. Furthermore, DCGD can be easily combined with other strategies and applied to variants of PINNs.

Although we have presented a novel optimization algorithm, DCGD, to address challenging issues in PINNs, there still remain some interesting and important questions. For instance, one could design a more powerful DCGD specification within the dual cone region that goes beyond the projection, average, and center techniques. Also, while we mainly consider multi-objective optimization for PINNs, future work can focus on more general and complex types of multi-task learning problems.

References

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Erik Laurin Strelow, Alf Gerisch, Jens Lang, and Marc E. Pfetsch. Physics informed neural networks: A case study for gas transport problems. Journal of Computational Physics , 481:112041, 2023. ISSN 0021-9991.

Pushan Sharma, Wai Tong Chung, Bassem Akoush, and Matthias Ihme. A Review of Physics-Informed Machine Learning in Fluid Mechanics. Energies , 16(5):2343, 2023.

Peter R. Wiecha, Arnaud Arbouet, Christian Girard, and Otto L. Muskens. Deep learning in nano-photonics: inverse design and beyond. Photon. Res. , 9(5):B182-B200, May 2021.

Mahmudul Islam, Md Shajedul Hoque Thakur, Satyajit Mojumder, and Mohammad Nasim Hasan. Extraction of material properties through multi-fidelity deep learning from molecular dynamics simulation. Computational Materials Science , 188:110187, 2021.

Jonthan D Smith, Zachary E Ross, Kamyar Azizzadenesheli, and Jack B Muir. HypoSVI: Hypocentre inversion with Stein variational inference and physics informed neural networks. Geophysical Journal International , 228(1):698-710, 2022.

Yogesh Verma, Markus Heinonen, and Vikas Garg. ClimODE: Climate Forecasting With Physics-informed Neural ODEs. In The Twelfth International Conference on Learning Representations , 2024.

Yuyan Ni, Shikun Feng, Wei-Ying Ma, Zhi-Ming Ma, and Yanyan Lan. Sliced Denoising: A PhysicsInformed Molecular Pre-Training Method. arXiv preprint arXiv:2311.02124 , 2023.

Lei Yuan, Yi-Qing Ni, Xiang-Yun Deng, and Shuo Hao. A-PINN: Auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations. Journal of Computational Physics , 462:111260, 2022.

Guofei Pang, Lu Lu, and George Em Karniadakis. fPINNs: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing , 41(4):A2603-A2626, 2019.

Dongkun Zhang, Ling Guo, and George Em Karniadakis. Learning in modal space: Solving time-dependent stochastic PDEs using physics-informed neural networks. SIAM Journal on Scientific Computing , 42(2): A369-A665, 2020.

Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. hp-VPINNs: Variational physicsinformed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering , 374:113547, 2021.

Ameya D Jagtap and George E Karniadakis. Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations. In AAAI spring symposium: MLPS , volume 10, 2021.

Benjamin Wu, Oliver Hennigh, Jan Kautz, Sanjay Choudhry, and Wonmin Byeon. Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations. In International Conference on Computational Science , pages 372-379. Springer, 2022.

Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks. International Conference on Learning Representations , 2024.

Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks. In Advances in Neural Information Processing Systems , 2023.

Junwoo Cho, Seungtae Nam, Hyunmo Yang, Seok-Bae Yun, Youngjoon Hong, and Eunbyung Park. Separable Physics-Informed Neural Networks. In Advances in Neural Information Processing Systems , 2023.

Jihun Han and Yoonsang Lee. Hierarchical learning to solve partial differential equations using physicsinformed neural networks. arXiv preprint arXiv:2211.08064v2 , 2023.

Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Computer Methods in Applied Mechanics and Engineering , 393:114823, 2022.

Hwijae Son, Sung Woong Cho, and Hyung Ju Hwang. Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs). Neurocomputing , page 126424, 2023.

Chuwei Wang, Shanda Li, Di He, and Liwei Wang. Is L 2 Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? In Advances in Neural Information Processing Systems , volume 35, pages 8278-8290, 2022.

Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404 , 2022.

Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering , 403:115671, 2023.

Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling. In International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 7264-7302. PMLR, 23-29 Jul 2023.

Arnaud Vadeboncoeur, Ömer Deniz Akyildiz, Ieva Kazlauskaite, Mark Girolami, and Fehmi Cirak. Fully probabilistic deep models for forward and inverse problems in parametric PDEs. Journal of Computational Physics , 491:112369, 2023.

Arnaud Vadeboncoeur, Ieva Kazlauskaite, Yanni Papandreou, Fehmi Cirak, Mark Girolami, and Ömer Deniz Akyildiz. Random Grid Neural Processes for Parametric Partial Differential Equations. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 34759-34778, 23-29 Jul 2023.

Johannes Müller and Marius Zeinhofer. Achieving High Accuracy with PINNs via Energy Natural Gradient Descent. In International Conference on Machine Learning , pages 25471-25485. PMLR, 2023.

Johannes Müller and Marius Zeinhofer. Position: Optimization in SciML Should Employ the Function Space Geometry. In Forty-first International Conference on Machine Learning , 2024.

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 26548-26560, 2021.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055-A3081, 2021.

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, 2022.

Sophie Steger, Franz M. Rohrhofer, and Bernhard C Geiger. How PINNs cheat: Predicting chaotic motion of a double pendulum. In The Symbiosis of Deep Learning and Differential Equations II , 2022.

Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence , 2022.

Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks. In International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39702-39721. PMLR, 23-29 Jul 2023.

Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin, and Noseong Park. DPM: A novel training method for physics-informed neural networks in extrapolation. In AAAI Conference on Artificial Intelligence , number 9, pages 8146-8154, 2021.

Bahador Bahmani and WaiChing Sun. Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings. arXiv preprint arXiv:2107.11496 , 2021.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Task Learning. In Advances in Neural Information Processing Systems , volume 33, pages 5824-5836, 2020.

Xiaojian Li, Yuhao Liu, and Zhengxian Liu. Physics-informed neural network based on a new adaptive gradient descent algorithm for solving partial differential equations of flow problems. Physics of Fluids , 35(6), 2023.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization, 2018.

Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique , 350(5):313-318, 2012.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-Averse Gradient Descent for Multitask learning. In Advances in Neural Information Processing Systems , volume 34, pages 18878-18890, 2021.

Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations , 2021.

Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-Task Learning as a Bargaining Game. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 16428-16446. PMLR, 17-23 Jul 2022.

Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent Component Alignment for Multi-Task Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20083-20093, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Harold M. Hochman and James D. Rodgers. Pareto optimal redistribution. The American economic review , 59(4):542-557, 1969.

Lukas Fesser, Richard Qiu, and Luca D'Amico-Wong. Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks. arXiv preprint arXiv:2306.09478v2 , 2023.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, et al. PINNacle: A comprehensive benchmark of physics-informed neural networks for solving PDEs. arXiv preprint arXiv:2306.08827 , 2023.

A Proofs for Section 4

Proof of Theorem 4.2. Recall that ϕ t is the angle between ∇L r ( θ t ) and ∇L b ( θ t ) , and R = ∥∇L r ( θ t ) ∥ ∥∇L b ( θ t ) ∥ at each iteration t . Consider a cone K t , defined as

Case (i). Suppose that ⟨∇L b ( θ t ) , ∇L r ( θ t ) ⟩ ≥ 0 . Observe that

Therefore, by the definition of the dual cone, we have ∇L ( θ t ) ∈ K ∗ t .

Case (ii). Suppose that ⟨∇L b ( θ t ) , ∇L r ( θ t ) ⟩ < 0 and -cos( ϕ t ) ≤ R ≤ -1 cos( ϕ t ) . By multiplying ∥∇L b ∥∥∇L r ∥ to -cos( ϕ t ) ≤ R , we get

On the other hand, by multiplying ∥∇L b ( θ t ) ∥ 2 cos( ϕ t ) to R ≤ -1 cos( ϕ t ) , we have

Therefore, we conclude that if ⟨∇L b ( θ t ) , ∇L r ( θ t ) ⟩ < 0 , then -cos( ϕ t ) ≤ R ≤ -1 cos( ϕ t ) is equivalent to ∇L ( θ t ) ∈ K ∗ t .

Proof of Proposition 4.3. Recall that

It is enough to show that we have ⟨ g, ∇L r ( θ t ) ⟩ ≥ 0 and ⟨ g, ∇L b ( θ t ) ⟩ ≥ 0 for any g ∈ G t . By the definition of G t , there exists c 1 , c 2 ≥ 0 such that g = c 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b for all g ∈ G t . One can easily check that

where ϕ t is the angle between ∇L r ( θ t ) and ∇L b ( θ t ) . One can derive that ⟨ g, ∇L b ( θ t ) ⟩ ≥ 0 in the same manner. Therefore, we conclude that G t ⊂ K ∗ t .

Proof of Theorem 4.5. Let ϕ t be the angle between ∇L r ( θ t ) and ∇L b ( θ t ) , and ψ t be the angle between g dual t and ∇L ( θ t ) at the t -th iteration. Note that θ t +1 = θ t -λg dual t where g dual t satisfies the conditions (i), (ii) of Theorem 4.5.

First of all, DCGD algorithm reaches a Pareto-stationary point if ϕ t = -1 by Definition 4.4, at which the optimization process is stopped.

Otherwise, we first observe from the differentiability and L -Lipschitz continuity condition of ∇L ( · ) for all x, y ∈ R d :

where we have used Cauchy-Schwarz inequality for the third inequality. Using Eq. (A.4) and Conditions (i), (ii) of Theorem 4.5, one calculates that for λ ≤ 1 2 L ,

By using telescoping sums, we further obtain

Proof of Proposition 4.6. Note that g c t is the angle bisector of ∇L r ( θ t ) and ∇L b ( θ t ) . From the formula of vector projection, g dual t of DCGD (Center) is the projection of ∇L ( θ t ) on to g c t . Thus, it is enough to show that g c t is included in G t .

We observe that which yields

and

Then, by defining c 1 = 1 ∥∇L b ( θ t ) ∥ (1 -cos( ϕ t )) and c 2 = 1 ∥∇L r ( θ t ) ∥ (1 -cos( ϕ t )) , one can easily see that

That is, g c t can be expressed as c 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b for some c 1 , c 2 ≥ 0 . Therefore, g c t is in G t .

Proof of Corollary 4.7. We will show that g dual t of each DCGD algorithm satisfies the conditions (i), (ii) of Theorem 4.5. Three algorithms are summarized in Algo. 2, Algo. 3, and Algo. 4. We note that a conflict threshold α is introduced as a stopping condition for DCGD algorithms, as they can reach a Pareto-stationary point characterized by ϕ t = π . That is, the algorithm stops when the parameter converges close to a Pareto-stationary point such that π -α < ϕ t ≤ π . Here, we assume α ≥ 0 is fixed.

1. DCGD (Projection): Note that it is trival to show that g dual t = ∇L ( θ t ) , when ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ ≥ 0 , satisfies the conditions (i), (ii). Thus, we focus on the case when ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ < 0 .

First of all, we need to show the condition (i)

which is equivalent to that ∥∇L ( θ t ) ∥ ≥ ∥ g dual t -∇L ( θ t ) ∥ . Using Eq. (A.6), one directly calculates that

In the same manner, we have ∥ 2 ∇ t L ∥L ⊥ b -∇L ( θ t ) ∥ 2 = ∥∇L ( θ t ) ∥ 2 . Since g dual t is chosen in G t , specifically c 1 = 1 , c 2 = 0 or c 1 = 0 , c 2 = 0 , we can write

where we have used c 1 + c 2 = 1 for obtaining the last inequality. Therefore, the condition (i) is satisfied.

We further suppose that ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ < 0 . Then, g dual t = ∇ t L ∥L ⊥ b . Let ϕ t be the angle between ∇L b ( θ t ) and ∇L r ( θ t ) , and ψ t be the angle between g dual t and ∇L ( θ t ) . Note that ϕ t ≤ π -α where α is conflict threshold. Otherwise, the algorithm stops when π -α < ϕ t ≤ π (see Algo. 2). Then, since ψ t = ϕ t -π 2 , we have

Thus, by choosing M = cos ( π 2 -α ) , the condition (ii) is satisfied. We repeat the same analysis for the case when ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ < 0 .

DCGD (Average): Similarly to DCGD (Projection), we focus on the case where DCGD (Average) specifies c 1 = c 2 = 1 2 , given by

when ∇L ( θ t ) / ∈ K ∗ t . Eq A.10 with c 1 = c 2 = 1 / 2 directly leads to

implying that the condition (i) is satisfied.

Next, suppose ⟨∇L ( θ t ) , ∇L b ( θ t ) ⟩ < 0 . Then, the condition (ii) is satisfied with M = 1 2 cos ( π 2 -α ) since

When ⟨∇L ( θ t ) , ∇L r ( θ t ) ⟩ < 0 , the condition (ii) is also satisfied with M = 1 2 cos ( π 2 -α ) .

DCGD (Center): the updated vector of DCGD (Center) is given by

where g c t = ∇L b ( θ t ) ∥∇L b ( θ t ) ∥ + ∇L r ( θ t ) ∥∇L r ( θ t ) ∥ . Since g dual t is the angle bisector of ∇L r ( θ t ) and ∇L b ( θ t ) (see Proof of Proposition 4.6), ψ t , the angle between ∇L ( θ t ) and g dual t , is less or equal to ϕ t / 2 , i.e., ψ t ≤ ϕ t 2 ≤ π 2 . From the fact that g dual t is the projection of ∇L ( θ t ) onto g c t , we have

and

Consequently, the conditions (i) and (ii) are satisfied for DCGD (Center).

Remark A.1 . Suppose that one employs a decaying scheme for the conflict threshold α t such that α t = O ( t -γ ) with where 0 ≤ γ < 1 , for example, α t = t -γ . In this case, the convergence rate of the DCGD algorithm to a stationary point becomes O ( 1 T 1 -γ ) , as M in condition (ii) may depend on the conflict threshold α t . For all our experiments, we set α to be fixed.

B Unification of MTL algorithms within the DCGD framework

In this section, we prove that several MTL algorithms can be understood as special cases of the DCGD framework under the PINN's formulation.

Proof. 1. MGAD [40]: The updated gradient g MGDA t of MGDA is defined by selecting the minimumnorm element from the convex combinations of ∇L r ( θ t ) and ∇L b ( θ t ) if there is gradient conflict ⟨∇L r ( θ t ) , ∇L b ( θ t ) ⟩ < 0 :

One can easily show that ⟨ g MGDA t , g MGDA t ⟩ = ⟨ g MGDA t , ∇L r ( θ t ) ⟩ = ⟨ g MGDA t , ∇L b ( θ t ) ⟩ ≥ 0 . Thus, g MGDA ∈ K ∗ .

PCGrad [37]: PCGrad uses the same update direction with DCGD (Average) when ⟨∇L r ( θ t ) , ∇L b ( θ t ) ⟩ < 0 ,

and takes ∇L r ( θ t ) + ∇L b ( θ t ) when ⟨∇L r ( θ t ) , ∇L b ( θ t ) ⟩ ≥ 0 . The latter case is also contained in K ∗ t . Therefore, g PCGrad t ∈ K ∗ t .

Nash-MTL [43]: Nash-MTL considers a Nash bargaining solution to balance the loss gradients. The update gradient g Nash-MTL t is be defined by

where G t = [ ∇L r ( θ t ) , ∇L b ( θ t )] . We can find the solution v t satisfying Eq. (B.2) as following. By letting v t = [ v 1 v 2 ] , we have

which is equivalent to

Therefore, we can derive v 2 = ∥∇L r ( θ t ) ∥ ∥∇L b ( θ t ) ∥ v 1 . Substituting v 2 = ∥∇L r ( θ t ) ∥ ∥∇L b ( θ t ) ∥ v 1 back into the first equation of Eq. (B.3) leads to

where ϕ t is the angle between ∇L r ( θ t ) and ∇L b ( θ t ) . Thus, the update gradient g Nash-MTL t has same direction with DCGD (Center). That is, g Nash-MTL t ∈ K ∗ t .

C Experimental details

C.1 Software and hardware environments

We conduct all experiments with PYTHON 3.10.9 and PYTORCH 1.13.1, CUDA 11.6.2, NVIDIA Driver 510.10 on Ubuntu 22.04.1 LTS server which equipped with AMD Ryzen Threadripper PRO 5975WX, NVIDIA A100 80GB and NVIDIA RTX A6000.

C.2 Toy example

We slightly modify the toy example in [41] to show our proposed method can expand the region of initial points that converge to the Pareto set. Consider the following loss functions with θ = ( θ 1 , θ 2 ) ∈ R 2 :

The landscape and contour map of above loss function are shown in Figure 8. The Pareto set is highlighted in gray in Figure 8b. We solve the above problem using ADAM, DCGD (Projection), DCGD (Average), DCGD (Center) for 100,000 epochs with different initial points. The initial points are selected as 1 , 600 uniform grid points within [ -10 , 10] × [ -10 , 10] . Then, we mark with a red dot the point at which the optimizer fails to converge to the Pareto set.

C.3 Details for Figure 2, Figure 1, and Figure 5

In this experiment, we use the 7 -layer fully connected neural network with 20 neurons per layer and a hyperbolic tangent activation function. We train PINN models using SGD with the learning rate of 0 . 01 for 10 , 000 epochs. In addition, 100 data points are sampled in boundaries and 10,000 points in the domain.

1

8

0

8

2

8

0

8

20

10

0

10

(a) Loss landscape

10

5

0

5

10

1

10

5

0

5

10

2

(b) Contour map

C.4 Details for Section 5.1

Benchmark equations We consider Helmholtz equation, viscous Burgers' equation, and KleinGordon equation as the benchmark equations.

The Helmholtz equation is described by

The solution is given by u ∗ ( x, y ) = sin( a 1 πx ) sin( a 2 πy ) where

In our experiment, we choose parameters: k = 1 , a 1 = 1 , a 2 = 4 as in [30].

The Viscous Burgers' equation is given by

where ν = 0 . 01 π .

The Klein-Gordon equation is

We set parameters to k = 3 , γ = 1 , T = 1 and the initial conditions, g 1 ( x ) = x, g 2 ( x ) = 0 for all x ∈ Ω following [30]. Then we can use the solution u ∗ ( t, x ) = x cos(5 πt ) + ( tx ) 3 where f ( t, x ) is derived by given equation .

We employ a 3-layer fully connected neural network with 50 neurons per layer and use the hyperbolic tangent activation function for all experiments in Section 5.1. At each iteration, 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points. We just randomly sample the points in the boundaries if there exists an analytic solution, otherwise the points were resampled from a pre-generated set for each iteration. More specifically, for the case

of Viscous Burger's equation, there is pre-determined 456 boundary points and we randomly sample in this set of points. We train PINNs for 50,000 epochs with Glorot normal initialization [48] using DCGD algorithms, ADAM [45], LRA [30], NTK [31], PCGrad [37], MultiAdam [34], and DPM [35].

We search for the initial learning rate among λ = { 10 -3 , 10 -4 , 10 -5 } and use a exponential decay scheduler with a decay rate of 0 . 9 and a decay step = 1 , 000 . For ADAM, we use the default parameters: β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 10 -8 as in [45]. For LRA, we set α = 0 . 1 , which is the best hyperparameter reported in [30]. For MultiAdam, we use β 1 , β 2 = 0 . 99 as recommended in [34]. For DPM, we test δ = { 10 -1 , 10 -2 , 10 -3 } , ϵ = { 10 -1 , 10 -2 , 10 -3 } , w = { 1 , 1 . 01 , 1 . 001 } .

To compute the effectiveness of various optimization algorithms, we evaluate the accuracy of the PINN solutions u ( · ; θ ) using the relative L 2 -error defined as:

where u ( · ) is the true solution and { x i } N i =1 is the set of test samples. Unless the equation has an analytic solution, we use the numerical reference solution for u ( x ) , which solved by finite element method [1].

In Table 3, we report the best and worst-case relative L 2 errors of each method across 10 independent trials (3 independent trials for two high-dimensional PDEs).

We plot the exact solution, PINN solution, and its error for each benchmark equation in Figures 10, 11, and 9

1

0

1

x

1

1.0

0.5

0.0

0.5

1.0

x

2

0.75

0.50

0.25

0.00

0.25

0.50

0.75

(a) Exact solution

1

0

1

x

1

1.0

0.5

0.0

0.5

1.0

x

2

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

(b) Prediction

1

0

1

x

1

1.0

0.5

0.0

0.5

1.0

x

2

0.000

0.005

0.010

0.015

0.020

0.025

0.030

(c) Absolute error

0.0

0.2

0.5

0.8

1.0

t

1.0

0.5

0.0

0.5

1.0

x

-0.8

-0.5

-0.2

0.0

0.2

0.5

0.8

(a) Exact solution

0.0

0.2

0.5

0.8

1.0

t

1.0

0.5

0.0

0.5

1.0

x

-1.0

-0.8

-0.5

-0.2

0.0

0.2

0.5

0.8

(b) Prediction

0.0

0.2

0.5

0.8

1.0

t

1.0

0.5

0.0

0.5

1.0

x

0.02

0.04

0.06

0.08

0.10

0.12

(c) Absolute error

0.0

0.5

1.0

t

0.0

0.2

0.4

0.6

0.8

1.0

x

0.5

0.0

0.5

1.0

1.5

(a) Exact solution

0.0

0.5

1.0

t

0.0

0.2

0.4

0.6

0.8

1.0

x

0.5

0.0

0.5

1.0

1.5

(b) Prediction

0.0

0.5

1.0

t

0.0

0.2

0.4

0.6

0.8

1.0

x

0.000

0.005

0.010

0.015

0.020

0.025

0.030

(c) Absolute error

High-dimensional equations We consider the following 3 -dimensional Helmholtz equation

The solution is given by u ∗ ( x, y ) = sin( a 1 πx ) sin( a 2 πy ) sin( a 3 πz ) where

Weemploy a 5-layer fully connected neural network with 128 neurons per layer and use the hyperbolic tangent activation function. At each iteration, 128 points are randomly sampled in boundaries and 500 times more points in the domain as the collocation points. We train PINNs for 30,000 epochs with Glorot normal initialization.

We use initial learning rate among λ = 10 -3 and use a exponential decay scheduler with a decay rate of 0 . 9 and a decay step = 1 , 000 .

For 5 -dimensional Heat equation, we follow the experiment setting in Hao et al. [49]. The PDE can be expressed as following:

where the geometric domain Ω = { x : || x || ≤ 1 } and

C.5 Details for Section 5.2

Double pendulum problem Consider the double pendulum which have two point mass pendulums with masses m 1 , m 2 , two rod with length l 1 , l 2 . Let θ 1 , θ 2 is the angle that the pendulums each make with the vertical and ∆ θ = θ 1 -θ 2 . Set the gravitational acceleration g = 9 . 81 . (see Figure 12)

Then the dynamics of double pendulum can be described by following nonlinear differential equation system with y = [ θ 1 , θ 2 ] T :

In this experiment, the initial conditions are θ 1 ( t 0 ) = θ 2 ( t 0 ) = θ 0 = 150 ◦ , ω 1 ( t 0 ) = ω 2 ( t 0 ) = 0 .

We replicate the experiment setup done in [32]. We use a six-layer feed-forward network of 30 neurons on each layer with the swish activation function. We train this model with ADAM optimizer with the default hyperparameters for 20,000 epochs. We also train the same model using DCGD (Center). Figure 13 shows the training curves of ADAM and DCGD.

0

2500

5000

7500

10000

12500

15000

17500

20000

epochs

10

3

10

2

10

1

10 0

L

train

L

F

1

L

F

2

L

IC

(a) ADAM

0

2500

5000

7500

10000

12500

15000

17500

20000

epochs

10

7

10

5

10

3

10

1

10 1

L

train

L

F

1

L

F

2

L

IC

(b) DCGD (Center)

Convection equation We train PINNsFormer in Liu et al. [15] to solve convection equation which can expressed as following:

Where β = 50 . we follow the default setting of [15] and train the model by 500 epochs.

chotic Kuramoto-Sivashinsky equation We use causal training in Wang et al. [22] to solve chaotic Kuramoto-Sivashinsky equation. We use 5 layers modifed-MLP with 64 neurons per layer. and train this model 50,000 epochs for each tolerance.

where α = 100 / 16 , β = 100 / 16 2 , γ = 100 / 16 4 .

0

2

4

6

x

0.0

0.2

0.4

0.6

0.8

1.0

t

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

(a) Exact solution

0

2

4

6

x

0.0

0.2

0.4

0.6

0.8

1.0

t

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

(b) Prediction

0

2

4

6

x

0.0

0.2

0.4

0.6

0.8

1.0

t

0.01

0.02

0.03

0.04

0.05

0.06

(c) Absolute error

0.00

0.25

0.50

0.75

1.00

x

1

0.0

0.2

0.4

0.6

0.8

1.0

x

2

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

(a) Exact solution

0.00

0.25

0.50

0.75

1.00

x

1

0.0

0.2

0.4

0.6

0.8

1.0

x

2

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

(b) Prediction

0.00

0.25

0.50

0.75

1.00

x

1

0.0

0.2

0.4

0.6

0.8

1.0

x

2

0.00

0.00

0.01

0.01

0.01

0.01

0.01

0.02

0.02

(c) Absolute error

Auxiliary-PINN: Nonlinear integro-differential equation A-PINN is a variant of PINNs, designed to solve integro-differential equations [9]. We apply our DCGD algorithms to A-PINN for solving the following nonlinear 2-dimensional Volterra IDE:

where the boundary conditions are u (0 , x ) = x , ∂u (0 ,x ) ∂t = sin( x ) , and u ( t, 0) = t sin( t ) , 0 ≤ t, x ≤ 1 . The analytic solution is u ∗ ( t, x ) = x + t sin( t + x ) with λ = 1 where g ( t, x ) is derived by given equation.

Within the framework of A-PINN, it converts the above integral equation into the following equation by representing integrals as auxiliary output variables:

where the new variables v and w satisfies the boundary condition v ( t, 0) = 0 , w (0 , x ) = 0 .

For A-PINNs, we employ a 3-layer fully connected neural network with 50 neurons per layer and a hyperbolic tangent activation function. For training, 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points in each epochs. We train A-PINN models for 5 , 000 epochs.

Separable PINN: 3-dimensional Helmholtz equation SPINN is a a novel architecture designed to effectively reduce the computational cost of PINNs, especially when addressing high-dimensional PDEs [17]. To test the performance of DCGD for SPINNs, we consider the following 3 -dimensional Helmholtz equation

The solution is given by u ∗ ( x, y ) = sin( a 1 πx ) sin( a 2 πy ) sin( a 3 πz ) where

We follow the optimal hyperparameter setting reported in [17]. For ADAM and DCGD (Center), the learning rate is 0 . 001 . The input points are resampled every 100 epochs. Regarding model architecture, we use the SOTA model, so called (SPINN + Modified MLP). We record the mean and standard deviation of relative L 2 errors from 3 independent trials in Table 4, indicating that the performance of SPINN can be significantly improved when trained with DCGD for a varying number of collocation points.

(a) Exact solution

(b) Prediction

(c) Absolute error

0.75

0.50

1.00

0.25

0.00

0.50

1.00

0.75

Z

0.!

0.C

0

0.75

1.00

0.50

0.25

0.25

0.00

1.00

0.75

0.50

0.!

0.C

0

1.C

0.75

0.50

1.00

0.25

0.25

0.00

0.50

0.75

1.00

0.!

0.C

1.C

1.08.18.58.28.08.26.56.74.00

0.25y

0.00

0.25

0.50

9.75

1.00

1.08.28.58.28.08.28.56.74.00

-0.25y

0.00

0.25

0.50

1.98.28.58.28.08.2.58.74.00

-0.25 y

0.00

0.25

0.50

1.00

0.75

0.50

1.00

0.75

0.50

1.00

0.75

0.50

D Supplemental results

D.1 Ablation study

In Section 4.3, we introduce three specific algorithms: DCGD (Projection), DCGD (Average), and DCGD (Center). We conduct an ablation study to investigate the impact of different updated gradient schemes within the dual cone region. More specifically, we compare the performance of these three algorithms on five benchmark equations. Detailed experimental settings can be found in Appendices C.4.

D.2 Computational cost

In this section, we acknowledge that our proposed method incurs higher computational costs due to the need for backpropagation for each individual loss. Nonetheless, through a comparison of training speeds, we empirically demonstrate that DCGD achieves superior performance with computational costs comparable to those of existing competitors.

E Pseudo codes of algorithms

This section provides pseudo codes for the proposed DCGD algorithms.

Firstly, DCGD (Projection) uses the projection of the total gradient ∇L ( θ t ) onto G t when ∇L ( θ t ) / ∈ K ∗ t . Otherwise, ∇L ( θ t ) is used. Then the update vector g dual t can be defined as follow:

Secondly, DCGD (Average) uses the the average of ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b when the total gradient is outside K ∗ t . Otherwise, ∇L ( θ t ) is used. The update vector g dual t of DCGD (Average) is defined as follow:

Thirdly, DCGD (Center) employs the following update vector g dual t , regardless of whether the total gradient ∇L ( θ t ) is included in K ∗ t :

Here, pseudo codes of these algorithms are summarized in Algorithms 2, 3, and 4. Note that we introduce a conflict threshold α as a stopping condition for DCGD algorithms, as they can reach a Pareto-stationary point characterized by ϕ t = π . That is, the algorithm stops when the parameter converges close to a Pareto-stationary point such that | cos( ϕ t ) -π | < α . Throughout our experiments, we set α = 10 -8 .

Algorithm 2 DCGD (Projection)

Require: learning rate λ , max epoch T , initial point θ 0 , gradient threshold ε , conflict threshold α for t = 1 to T do

DCGD algorithms can be easily combined with other optimizers or strategies thanks to its flexible framework. For example, one can design a DCGD algorithm combined with ADAM to leverage advantages of adaptive gradient methods, see Algo. 5. For our experiments, we consider the DCGD (center) combined with ADAM as the default. Algorithm 6 presents the psedo code for DCGD combined with a loss balancing method such as LRA and NTK.

Algorithm 3 DCGD (Average)

Require: learning rate λ , max epoch T , initial point θ 0 , gradient threshold ε , conflict threshold α for t = 1 to T do

break end if

g

1

t

=

dual

2

∇

t

L

∥∇L

⊥

else

Algorithm 4 DCGD (Center)

Require: learning rate λ , max epoch T , initial point θ 0 , gradient threshold ε , conflict threshold α for t = 1 to T do

break

θ

t

=

θ

t

-

1

-

λg

t

end for

Algorithm 5 DCGD with ADAM

Require: learning rate λ , max epoch T , betas β 1 , β 2 , DCGD operator DCGD ( · )

end for

Algorithm 6 DCGD with a loss balancing method

Require: learning rate λ , max epoch T , loss balancing operator LB ( · )

r

+

1

2

∇

t

L

∥∇L

⊥

b

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] Our abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] We have discussed the limitations of our work in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] We have stated the explicit assumptions and provide complete proofs for main results in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] The pseudo code of our algorithms can be found in Appendix E, and we have stated detailed instructions in Appendix C. We have also provided code to reproduce the results of main experiments in supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] We have provided our code to reproduce the results of main experiments in supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] All experimental setting is explained in Appendix C including hyperparameter choices, training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

We repeated the main experiments in several times and reported several statistics including min, max, and the standard deviations for our numerical results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] See Application C.1 for details of resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] The overall experiments followed the existing experimental setup which have cited, and the code and data are available for use under the MIT license. For CausalPINNs in [22], the code is available under the CC-BY-NC-SA 4.0 license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary:** The paper proposes viewing the optimization of PINNs as a multi-objective optimization problem, with boundary and residual terms as potentially conflicting objective functions. To address conflicting gradients in the optimization process, the dual cone of the cone generated by the gradients of the boundary and residual terms is designated as the set of non-conflicting update directions. Several algorithms that guarantee updates from this dual cone are proposed, and their empirical performance is assessed.\n\n**General Impression:** The manuscript is well-written, easy to follow, and generally of high quality. The numerical experiments are convincing and demonstrate the good performance of the proposed method. However, I disagree to a certain extent with the multi-objective viewpoint for the optimization of PINNs. For well-posed forward problems, the boundary loss and the interior loss are not really conflicting losses—at least if we assume that the network ansatz is sufficiently expressive. I expand more on this issue below in the questions. Nevertheless, the preprint presents a convincing suite of numerical experiments that seem to benefit from the multi-objective optimization approach.\n\nSTRENGTHS:\n**Strengths:**\n\n1. Considering the dual cone of the interior and residual gradients is a simple and sound idea. The presentation is clear and understandable.\n2. The section on numerical results demonstrates the benefits of the proposed method and incorporates many recently proposed techniques for the optimization of PINNs.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n**Questions**\n\n1. It is unclear whether PINN optimization should be viewed as a multi-objective optimization problem. Please comment.\n2. I am missing a clarification on whether the proposed dual cone gradient descent is a completely novel idea, even in the field of multi-objective optimization, or if it is well-known in that field and its application to PINNs is novel. Please clarify and give references if appropriate.\n3. Could the authors comment on how their proposed method relates to loss reweighting strategies? It seems that the proposed approach could equivalently be understood as a loss reweighting strategy.\n4. Can the authors comment on the case of more than two loss functions.\n5. Have the authors considered measuring angles in function space instead of parameter space by using the differential of the parametrization map? Recently, optimization methods that employ function space geometry have shown promise for the optimization of PINNs (see [1, 2, 3]). This might lead to further improvement of the proposed method.\n\n**Details to question 1:**\nA fundamental question for me is whether to view PINNs as a multi-objective optimization problem or not. To illustrate this question, consider the following academic example of a PDE with homogeneous forcing and inhomogeneous boundary data:\n\n\\begin{align}\nNu &= 0 \\quad \\text{in }\\Omega,\n\\\\\n\\newline\nBu &= g \\quad \\text{on }\\partial\\Omega.\n\\end{align}\n\nAssume further that $N$ maps constant functions to zero (any PDE operator without a zeroth-order term). Assume a trial neural network is initialized representing (almost) the zero function (or any other constant function). Any change in the neural network will now likely increase the PDE residual (which is at its global minimum for any constant function). Furthermore, the increase in the PDE residual is necessary to obtain the correct solution. This might lead to a Pareto stationary point that has nothing to do with the sought-after solution.\n\n**References**\n\n[1] https://proceedings.mlr.press/v202/muller23b/muller23b.pdf\n\n[2] https://openreview.net/pdf?id=z9SIj-IM7tn\n\n[3] https://arxiv.org/pdf/2402.07318\n\nLIMITATIONS:\nSee questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper concerns with the training of physics-informed neural networks (PINNs). It observes that in a multi-objective setting a naive gradient update which decreases the total objective might not decrease every individual objective. This is used to explain the challenge of training PINNs which have a potentially conflicting PDE and a boundary-condition loss. As a remedy, the work proposes to find another descent direction which decreases both of the PINN losses. Three different methods are proposed and the superior \"center\" method is selected for more detailed empirical investigation. It demonstrates strong results relative to several existing PINN optimizers.\n\nSTRENGTHS:\nThe paper presents a clear central story that explains multi-objective optimization in terms of simple geometric concepts. The theoretical analysis in addition to the empirical evaluation is a welcome addition.\n\nWEAKNESSES:\nHowever, the significance of the theoretical contributions must be questioned. There are just two theorems, of which 4.2 is fairly trivial and 4.5 is unclear: why is $\\frac{1}{T+1}$ on both sides of Equation 4.2, and what does this equation even state? I cannot make sense of the provided intuition that the convergence rate is $\\mathcal{O}(1/T)$, i.e., the more iterations, the slower you converge. Even more so, it is unclear how this intuition follows from Eq. 4.2 in the first place: it says that the sum of gradient norms is upper-bounded by the gap from the initial to the optimal loss, up to some constant.\n\nThe work also overlooks some recent literature explaining the difficult behavior of PINNs in terms of poor conditioning and proposing strong optimization strategies, which to the best of my knowledge achieve state-of-the-art convergence. I would argue these ideas should be included in the related work and as baselines contributing to a much more complete story.\n- *De Ryck et al. An operator preconditioning perspective on training in physics-informed machine learning. ICLR 2024.*\n- *Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.*\n\nSeveral presentation choices could be improved, including minor mathematical and typing errors, odd wording, and superfluous parts (see suggestions below).\n\nThe work also overstates the practical success of PINNs: \"a prominent approach for solving PDEs\" (line 2), \"remarkable empirical performance\" (line 4), \"PINNs have achieved great success in a wide range of applications\" (line 8). This sets a misleading context for the practical significance of PINNs and by extension the presented work.\n- *Grossmann et al. Can Physics-Informed Neural Networks beat the Finite Element Method? IMA Journal of Applied Mathematics 2024.*\n\nQUESTIONS:\n- I would kindly ask the authors the clarify Theorem 4.5 in terms of the aforementioned concerns.\n- How do the aforementioned optimization strategies compare to your experiments? Do you foresee a way for your method to be compatible to these (similar to section 5.3)?\n- Does your method extend to more than two losses? Even in PINNs, there might be additional losses for data and initial conditions.\n- How important is the assumption of fixing $\\omega_r = \\omega_b = 1$? Do the other optimizers show improved performance for other choices of these loss weights?\n\nLIMITATIONS:\nThe limitations are discussed rather sparingly and should also include that the method currently applies to just two losses. It should also be noted that the performed experiments are closer to toy than practical problems. \n\n**Errors and suggestions**\n- Line 59: It would be helpful if you briefly described what each of these three approaches do, instead of only listing examples.\n- Line 73: $d$ is used for both the dimension of the domain as well as the parameter space.\n- Eq. 2.2 and line 88: $\\omega_u$ should be $\\omega_b$\n- Line 79: should this be an equality?\n- I do not think writing out algorithm 1 adds much value as this is precisely gradient descent but with a different gradient.\n- Line 170: should there be an argmin or arginf instead of inf? Or alternatively $\\mathcal{L}(\\theta*)$ instead of $\\theta*$\n- Lines 229-240 and Figure 6 might better belong to the appendix as it detracts attention from the main story.\n- Line 185: \"presents\" -> \"present\"\n- Line 221: \"ADAM\" -> \"Adam\"\n- A.4: the first inequality should be an equality\n- Adjust some odd wording, e.g. \"mysterious challenges\" (line 44), \"harmoniously\" (line 48)\n- I suggest to moderate the claims about the practical success of PINNs (see weaknesses for concrete examples).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
In this paper, it is identified that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, the authors propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, the convergence properties of DCGD algorithms in a non-convex setting are analyzed. On a variety of benchmark equations, DCGD outperforms other optimization algorithms in terms of various evaluation metrics, achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs.

STRENGTHS:
1. Theoretical analysis is provided, including the necessary and sufficient conditions under which the total gradient falls within the dual cone region, and the convergence rate.
2. Superior performance.

WEAKNESSES:
1. It is better to provide the visualization of PINN predictions and compare with exact solutions, especially for the failure modes of PINNs.
2. Training timing is not reported.

QUESTIONS:
Line 277, "Failure model" vs. "Failure modes"?

LIMITATIONS:
The proposed Dual Cone Gradient Descent (DCGD) optimization method for PINNs may still be trapped in local minima.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPINNs have become commonplace in both scientific computing and machine learning communities and have found widespread applications. There have been many modifications to various aspects of PINNs, such as the loss functions, initialisations, initial and boundary condition representations, learning algorithms and network architectures. \nA major problem with learning effective solutions through PINNs relative to other learning tasks is their difficulty in converging to the actual solution due to the complex nature of the underlying physics and equations involved and the corresponding loss landscape. \nThe authors have provided a comprehensive survey of the previous works, making the unaddressed problems and goals very clear. A novel algorithmic framework based on convex optimization and geometric analysis is presented, which demonstrates superior performance on many tasks. \nThe contributions of this paper are multifold, one being a new optimization algorithm while the other being a unified, open and flexible framework to adapt, modify and integrate the current technique with previous and future techniques.\nThe paper is well written with enough information on the theoretical and mathematical underpinnings, experimental setup, implementation details, results and justifications.\n\nSTRENGTHS:\nAlthough the concept of dual cone is well known in the optimization domain, specifically in the convex optimization area, the authors utilise and combine existing techniques from optimization theory in a novel way to develop a new algorithmic framework, which they refer to as DCGD. \n\nThe authors propose the DCGD algorithm which involves picking a gradient from the Dual Cone region formed by the two losses (boundary loss and residual loss) of PINNs and also theoretically prove that it converges to a pareto-stationary point. Thus, along a new algorithm, theoretical guarantees for convergence are provided, which are very crucial for the goal of improving the training convergence of PINNs.\n\nWith a suitable candidate being chosen as the new gradient from an explicitly defined space (called here as $G_t$), multiple variants of the DCGD algorithm can be created. The DCGD can also be combined with many existing optimization techniques, such as Adam Optimizer and Neural Tangent Kernel (NTK). \n\nThis work introduces a fairly novel optimization technique which has huge potential for applications in PINNs and multi-objective optimization. The authors have provided strong justification for the generality of their work by showing that most previous MTL approaches are special cases of DCGD. The general, modular and extensible nature of the work paves the way to significantly accelerate further research in various allied areas.\n\nThe supplementary material includes the datasets and code for all the experimental details mentioned in the paper and the appendices. I have glanced through the code and data files provided but have not personally verified the results by running the code. (might update here if I get to install and run the code provided)\n\nWEAKNESSES:\nThe authors have provided enough empirical and theoretical evidence for their proposed DCGD method, though since this work proposes a theoretical and algorithmic framework, albeit it would be notable to have a section on comparing the computational complexity and qualitative terms with existing algorithms. I would say a short section in Appendix D/E would go well.\n\nSince the problem at hand is of multi-objective optimization which overlaps with multi-task learning problems and PINNs are a natural suitable choice, it would be very helpful to see the results of this approach on PINOs (Physics Informed Neural Operators). There is no comparison (even qualitative) of DCGD on PINOs (There are PINOs which are trained only on residual loss, but here I'm specifically referring to PINOs which include both residual and boundary loss for being relevant to DCGD). Was this not considered or omitted, or is it expected to show improvement in performance, as PINNs have shown? (Personally, I believe it should show improvements as the authors have already demonstrated superior performance on PINNformer with DCGD)\n\n\nPresentations/Grammar/Typos (I do understand that the authors would be well aware of these and would rectify these in the pre-print or camera-ready versions. However, I would like to mention them here together to aid them. Some might have been missed; might update if needed)\n\nEquation 2.2: The weight of the boundary loss is mentioned as $\\omega_u$ and also on line 80, whereas it is mentioned as $\\omega_b$ (which seems more appropriate) on line 79\n\nSection 4.3: (This is a nitpick and can be conveniently ignored)\n$g_t^c$ could be written on the same line as eq. 4.3 or as a newline as eq. 4.4 similar to how equation E.3 is written\n\nSection 5.2: Is the title intended to be “P(I)DEs” or “PDEs”? I believe this is just a typo\n\nAppendix A, line 769: Equations A.8 and A.9 could be mentioned here (as A.4 and A.5) and then referenced in the Proof of Proposition 4.6 section. I believe this will greatly enhance readability\n\nAppendix A: Proof of Theorem 4.2 line 757: The residual loss is repeated twice $\\nabla\\mathcal{L}_r(\\theta_t)$ is repeated twice, it should be both boundary and residual loss, $\\nabla\\mathcal{L}_r(\\theta_t)$ and $\\nabla\\mathcal{L}_b(\\theta_t)$\n\nAppendix A, line770: \"One can derive that …\" should we include the boundary loss term instead of residual loss as the derivation with residual loss is already shown in line769 i.e it should be $\\langle g, \\nabla L_b (\\theta_t) \\rangle \\geq 0$\n\nAppendix A, line 781: For the sake of notational consistency, could you replace \"condition (2)\" with \"condition (ii)\"\n\nAppendix A, line 785: (nitpick) The subscript t is omitted in the boundary loss and included in the residual loss ($\\nabla\\mathcal{L}_r(\\theta_t)$ and $\\nabla\\mathcal{L}_b(\\theta)$). I suggest the subscript \"t\" be included at both the places for notational consistency\n\nAppendix A, line 818, Proof of Corollary 4.7:  3. DCGD (Center): “vector” instead of “veoctr”\n\nQUESTIONS:\nI have gone through the previous works mentioned in detail, and does seems like the problem of multi-objective optimization or even optimization, specifically in the case of PINNs, has not been looked at from the perspective of Dual Cones. Could the authors add some more works related to Dual Cone-based techniques to the previous/related works section?\n\nTypos and presentation issues are already covered in the weaknesses section; however, here, I would like to mention a few pointers where I would like to seek clarification. (This is more from a clarification perspective for me individually than a comment on the work, though I believe these clarifications will help the broader community)\n\nAppendix A, line 779, Proof of Theorem 4.5: In the derivation of line779, from the first step to the next, should there be an equality sign in place of the first inequality sign as we are just distributing the dot product over addition on the first term $\\nabla \\mathcal{L}(y+t(x-y))$? (i.e. the first of the 3 inequality signs should be an equality instead)\n\nAppendix A line 797: Is the expression $\\left| \\cos(\\phi_t) - \\pi \\right| < \\alpha$ is correct and as intended? The inequalities on lines 810 and 811 and the threshold criterion in the algorithms in Appendix E ($\\pi - \\alpha < \\phi_t \\leq \\pi$) point otherwise. I believe there should not be the cosine of $\\phi$; rather, it should just be the angle/phase difference $\\phi$.\nI have a similar comment for line 984, as the same expression is used.\n\nAppendix C.5 line 925: Is the statement complete? Is it intentionally left this way?\n\nLIMITATIONS:\nI am appreciative of the authors who have mentioned (and also acknowledged in the checklist) a few limitations of the work in section 6 this is more from a future work perspective than from a limitations of the work presented as part of this paper. The limitations of the current work are not very evident from the present text.\n\nIt would be further helpful if they could include a more detailed section on where this cannot be applied or other multi-objective scenarios where the DCGD optimization framework is not as effective as other previous techniques.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}