Dual Cone Gradient Descent for Training Physics-Informed Neural Networks

Authors: youngsik  hwang
dong-young  lim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 14:39:54

Abstract: Physics-informed neural networks (PINNs) have emerged as a prominent approach for solving partial differential equations (PDEs) by minimizing a combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs often fail to generate reasonable solutions, and such pathological behaviors remain difficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, we propose a novel optimization framework, *Dual Cone Gradient Descent* (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in a non-convex setting. On a variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms of various evaluation metrics. In particular, DCGD achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD can be further improved by combining it with popular strategies for PINNs, including learning rate annealing and the Neural Tangent Kernel (NTK).

Main paper: 
 introduction: 
 Physics-informed Neural Networks (PINNs) proposed in Raissi et al. [1] have created a new paradigm in deep learning for solving forward and inverse problems involving partial differential equations (PDEs). The key idea of PINNs is to integrate physical constraints, governed by PDEs, into the loss function of neural networks. This is in turn equivalent to finding optimal parameters for the neural network by minimizing a loss function that combines boundary loss and PDE residual loss. Thanks to their strong approximation ability and mesh-free advantage, PINNs have achieved great success in a wide range of applications [2] [3] [4] [5] [6] [7] [8] .Building upon this success, the applications of PINNs have been extended to solve other functional equations, including integro-differential equations [9] , fractional PDEs [10] , and stochastic PDEs [11] . Moreover, numerous variants of PINNs have been developed to enhance their computational efficiency and accuracy via domain decomposition methods [12, 13] , advanced neural network architectures [14] [15] [16] [17] [18] , modified loss functions [19] [20] [21] , different sampling strategies [22] [23] [24] , andNote that it is trival to show that g dual t = ∇L(θ t ), when ⟨∇L(θ t ), ∇L b (θ t )⟩ ≥ 0, satisfies the conditions (i), (ii). Thus, we focus on the case when ⟨∇L(θ t ), ∇L b (θ t )⟩ < 0.First of all, we need to show the condition (i)2⟨∇L(θ t ), g dual t ⟩ -∥g dual t ∥ 2 = ∥∇L(θ t )∥ 2 -∥g dual t -∇L(θ t )∥ 2 ≥ 0,which is equivalent to that ∥∇L(θ t )∥ ≥ ∥g dual t -∇L(θ t )∥. Using Eq. (A.6), one directly calculates that∥2∇ t L ∥L ⊥ r -∇L(θ t )∥ 2 = ∇L(θ t ) -⟨∇L(θ t ), ∇L r (θ t )⟩ ∇L r (θ t ) ∥∇L r (θ t )∥ 2 2 = ∥∇L(θ t )∥ 2 .In the same manner, we have∥2∇ t L ∥L ⊥ b -∇L(θ t )∥ 2 = ∥∇L(θ t )∥ 2 . Since g dual t is chosen in G t , specifically c 1 = 1, c 2 = 0 or c 1 = 0, c 2 = 0, we can write ∥g dual t -∇L(θ t )∥ = c 1 ∇L ∥L ⊥ r (θ t ) + c 2 ∇L ∥L ⊥ b (θ t ) -∇L(θ t ) ≤ c 1 ∇L ∥L ⊥ r (θ t ) + c 2 ∇L ∥L ⊥ b (θ t ) - c 1 + c 2 2 ∇L(θ t ) + c 1 + c 2 2 -1 ∇L(θ t ) ≤ c 1 2 2∇L ∥L ⊥ r (θ t ) -∇L(θ t ) + c 2 2 2∇L ∥L ⊥ b (θ t ) -∇L(θ t ) + c 1 + c 2 2 -1 ∇L(θ t ) = c 1 2 ∥∇L(θ t )∥ + c 2 2 ∥∇L(θ t )∥ + c 1 + c 2 2 -1 ∥∇L(θ t )∥ = c 1 + c 2 2 + c 1 + c 2 2 -1 ∥∇L(θ t )∥ (A.10) = ∥∇L(θ t )∥.where we have used c 1 + c 2 = 1 for obtaining the last inequality. Therefore, the condition (i) is satisfied.We further suppose that ⟨∇L(θ t ), ∇L b (θ t )⟩ < 0. Then, g dual t = ∇ t L ∥L ⊥ b . Let ϕ t be the angle between ∇L b (θ t ) and ∇L r (θ t ), and ψ t be the angle between g dual t and ∇L(θ t ). Note that ϕ t ≤ π -α where α is conflict threshold. Otherwise, the algorithm stops when π -α < ϕ t ≤ π (see Algo. 2). Then, since ψ t = ϕ t -π 2 , we have∥g dual t ∥ = ∥∇ t L ∥L ⊥ b ∥ = ∥∇L(θ t )∥ cos(ψ t ) = ∥∇L(θ t )∥ cos ϕ t - π 2 ≥ ∥∇L(θ t )∥ cos π 2 -α .Thus, by choosing M = cos π 2 -α , the condition (ii) is satisfied. We repeat the same analysis for the case when ⟨∇L(θ t ), ∇L b (θ t )⟩ < 0.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Our abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries: 
 Notation. The Euclidean scalar product is denoted by ⟨•, •⟩, with ∥ • ∥ standing for the Euclidean norm (where the dimension of the space may vary depending on the context). For a subspace W of a vector space V , its orthogonal complement W ⊥ is defined asW ⊥ := {v ∈ V |⟨u, v⟩ = 0, u ∈ W }.For a vector v ∈ V , the projection of v on a nontrivial subspace W is denoted by v ∥W . Unless otherwise specified, V represents R d throughout the paper. Related Works. Among various research directions in PINNs, we focus on reviewing optimization strategies for PINNs. These can be broadly categorized into three main approaches: adaptive loss balancing, gradient manipulation, and Multi-Task Learning (MTL). As an example of adaptive loss balancing algorithms, Wang et al. [30] proposed a learning rate annealing (LRA) algorithm that balances the loss terms by utilizing gradient statistics. Wang et al. [31] utilized the eigenvalues of the Neural Tangent Kernel (NTK) to address the disparity in convergence rates among different losses of PINNs. For gradient manipulation algorithms, the Dynamic Pulling Method (DPM) was proposed in [35] to prioritize the reduction of the PDE residual loss. In [36] , the authors used the PCGrad algorithm, proposed in [37] , for training PINNs to address multi-task learning challenges. Li et al. [38] developed an adaptive gradient descent algorithm (AGDA) that resolves the conflict by projecting boundary condition loss gradient to the normal plane of the PDE residual loss gradient. Yao et al. [34] recently developed MultiAdam, a scale-invariant optimizer, to mitigate the domain scaling effect in PINNs. Another important line of gradient manipulation involves Multi-Task Learning (MTL) algorithms, which optimize a single model to perform multiple tasks simultaneously [39, 40, 37, [41] [42] [43] [44] . We will discuss that several MTL algorithms can be unified within the proposed DCGD framework.Physics-Informed Neural Networks. Let Ω ⊆ R D be a domain and ∂Ω be the boundary of Ω. We consider the following nonlinear PDEs:N [u](x) = f (x), x ∈ Ω B[u](x) = g(x), x ∈ ∂Ω (2.1)where N and B denote a nonlinear differential operator and a boundary condition operator, respectively. We approximate u(x) by a deep neural network u(x; θ) parameterized by θ. To train the neural network, the framework of PINNs minimizes the total loss function L(θ), which is a weighted sum of PDE residual loss L r (θ) and boundary condition loss L b (θ), defined by:L(θ) := ω r L r (θ) + ω b L b (θ) with (2.2) L r (θ) := 1 N r Nr i=1 |N [u(•; θ)](x i r ) -f (x i r )| 2 , L b (θ) := 1 N b N b i=1 |B[u(•; θ)](x i b ) -g(x i b )| 2 ,where ω r , ω b ≥ 0 are weights of each loss term, {x i r } Nr i=1 denotes a set of collocation points that are randomly sampled in Ω, and {x i b } N b i=1 the boundary sample points. Here, we set ω r = ω b = 1 throughout the paper. We note that the training of PINNs falls into the category of multi-objective learning due to its structure of the loss function L(θ) in Eq. (2.2). This section investigates issues that are frequently observed during the training of PINNs in the context of multi-objective learning. The parameter for the PINN solution u(x; θ) is typically estimated by minimizing the total loss function L(θ) with a (stochastic) gradient descent method 2 :Similarly to DCGD (Projection), we focus on the case where DCGD (Average) specifies c 1 = c 2 = 1 2 , given byg dual t = 1 2 ∇ t L ∥∇L ⊥ r + ∇ t L ∥∇L ⊥ b ,when ∇L(θ t ) / ∈ K * t . Eq A.10 with c 1 = c 2 = 1/2 directly leads to∥g dual t -∇L(θ t )∥ ≤ ∥∇L(θ t )∥, implying that the condition (i) is satisfied. Next, suppose ⟨∇L(θ t ), ∇L b (θ t )⟩ < 0. Then, the condition (ii) is satisfied with M = 1 2 cos π 2 -α since ∥g dual t ∥ = 1 2 ∇ t L ∥L ⊥ r + ∇ t L ∥L ⊥ b ≥ 1 2 ∥∇ t L ∥L ⊥ b ∥ = 1 2 ∥∇L(θ t )∥ cos ϕ t - π 2 ≥ 1 2 cos π 2 -α ∥∇L(θ t )∥.When ⟨∇L(θ t ), ∇L r (θ t )⟩ < 0, the condition (ii) is also satisfied with M = 1 2 cos π 2 -α .3. DCGD (Center): the updated vector of DCGD (Center) is given by g dual t = ⟨g c t , ∇L(θ t )⟩ ∥g c t ∥ 2 g c t where g c t = ∇L b (θt) ∥∇L b (θt)∥ + ∇Lr(EQUATION∥g dual t ∥ = ∥∇L(θ t )∥ cos(ψ t ) ≥ ∥∇L(θ t )∥ cos ϕ t 2 ≥ ∥∇L(θ t )∥ cos π -α 2 .Consequently, the conditions (i) and (ii) are satisfied for DCGD (Center).Remark A.1. Suppose that one employs a decaying scheme for the conflict threshold α t such that α t = O(t -γ ) with where 0 ≤ γ < 1, for example, α t = t -γ . In this case, the convergence rate of the DCGD algorithm to a stationary point becomes O 1 T 1-γ , as M in condition (ii) may depend on the conflict threshold α t . For all our experiments, we set α to be fixed.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] We have discussed the limitations of our work in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
empirical observations and issues in training pinns: 
 θ t+1 = θ t -λ∇L(θ t ), t ∈ Nwhere ∇L(θ) is the gradient of the total loss function L(θ) with respect to θ. However, a careless adoption of standard gradient descent methods may lead to an incorrect solution, as reducing the total loss does not necessarily imply a decrease in both the PDE residual loss and boundary loss. This phenomenon is clearly illustrated in Figure 1 , which displays the curves of the total loss, PDE residual loss, and boundary loss over epochs for solving the viscous Burger's equation. Notably, while the total loss consistently decreases throughout the training, the PDE loss adversely increases.Conflicting and dominating gradients in PINNs. This issue is highly related with discrepancies in the direction and magnitude between two gradients of the PDE residual and boundary loss. Specifically, we define two gradients to be conflicting at the t-th iteration if they have a negative inner product value, i.e., π 2 < ϕ t ≤ π where ϕ t is the angle between ∇L r (θ t ) and ∇L b (θ t ). When there are conflicting gradients, parameter updates to minimize one loss function might increase the other, leading to an inefficient learning process such as oscillating between optimizing for the two loss functions and resulting in degraded solution quality [19] . Another problem arises when one gradient is much larger than the other, i.e., ∥∇L r (θ t )∥ ≪ ∥∇L b (θ t )∥ or ∥∇L r (θ t )∥ ≫ ∥∇L b (θ t )∥. The significant differences 3 in the magnitudes of gradients in PINNs might create a situation where the optimization algorithm primarily minimizes one loss function while neglecting the other. This often results in slow convergence and overshooting, as the smaller gradient, though neglected, may be more crucial in finding a better solution. To mitigate the imbalance in the gradients, loss balancing approaches to rescale the weights of each loss term have been proposed [30, 31] .To examine these challenges in training PINNs, we record cosine value of the angle between ∇L r and ∇L b , and the ratios of their magnitudes while training a PINN for the Helmholtz equation. Figure 2(a) shows that conflicting gradients are observed in about half of the total iterations. Moreover, we observe that the magnitude of the gradient of the PDE residual is several tens to hundreds of times larger than that of the boundary loss (See Figure 2 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] We have stated the explicit assumptions and provide complete proofs for main results in Appendix A.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
methodology: 
 In this section, we provide a geometric analysis to identify a dual cone region where both the PDE residual loss and the boundary loss can decrease simultaneously. Subsequently, we introduce a general framework for DCGD algorithms, ensuring that the updated gradient falls within this region.We then propose three specifications of DCGD algorithms: projection, average, and center. All proofs for main results in this section can be found in Appendix A.The concept of a dual cone plays a pivotal role in our DCGD algorithm. Formally, a dual cone is defined as a set of vectors that have nonnegative inner product values with a given cone. Definition 4.1. (Dual cone) Let K be a cone of R d . Then, the setK * = {y|⟨x, y⟩ ≥ 0 for all x ∈ K, y ∈ R d } is called the dual cone of K.For each iteration t, consider a cone denoted by K t , which is generated by rays of two gradients, ∇L r (θ t ) and ∇L b (θ t ):K t := {cx|c ≥ 0, x ∈ {∇L r (θ t ), ∇L b (θ t )}} .In the context of PINNs, the dual cone of K t , denoted by K * t , represents the set of gradient vectors where each vector is neither conflicting with the gradient of the PDE loss nor with the gradient of the boundary loss, i.e., for u ∈ K * t , ⟨u, ∇L r (θ t )⟩ ≥ 0 and ⟨u, ∇L b (θ t )⟩ ≥ 0. In other words, when the total gradient ∇L(θ t ) is in K * t (as depicted by the region of the red line in Figure 1 ), the standard gradient descent taking the direction ∇L(θ t ) will decrease both the PDE and boundary losses for a suitable step size. On the other hand, if ∇L(θ t ) / ∈ K * t (the region indicated by the blue line in Figure 1 ), one of the two losses will adversely increase even with sufficiently small step sizes. This indicates that the training process of PINNs can significantly vary depending on whether the total gradient belongs to the dual cone region. The following theorem establishes the necessary and sufficient conditions under which the total gradient falls within the dual cone region in terms of the angle and relative magnitude between the gradients of the PDE residual and boundary loss. Theorem 4.2. Suppose that ∇L r (θ t ) and ∇L b (θ t ) are given at each iteration t. Let ϕ t be the angle between ∇L r (θ t ) and ∇L b (θ t ), and R = ∥∇Lr(θt)∥ ∥∇L b (θt)∥ be their relative magnitude. Then,∇L(θ t ) ∈ K * t if and only if (i) ⟨∇L b (θ t ), ∇L r (θ t )⟩ ≥ 0 , or (ii) ⟨∇L b (θ t ), ∇L r (θ t )⟩ < 0 and -cos ϕ t ≤ R ≤ - 1 cos ϕ t .Theorem 4.2 provides a clear criterion for when conflicting and dominating gradients lead to adverse training in PINNs. For instance, the condition (ii) in Theorem 4.2 implies that the larger ϕ t (the more conflicting they are), even a slight difference in their magnitudes can result in adverse training.In particular, Theorem 4.2 quantifies the extent of problematic relative magnitude between the two Figure 3 : Visualization of dual cone region K * t and its subspace G t gradients, thereby clarifying the concept of dominating gradients, which has not been previously defined in the literature.Thus, our strategy aims to devise an algorithm that chooses the updated gradient within the dual cone region at each gradient descent step. For notational simplicity, we write ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b to represent ∇L(θ t ) ∥(∇Lr(θt)) ⊥ and ∇L(θ t ) ∥(∇L b (θt)) ⊥ , respectively. In particular, we are interested in a simple and explicit subspace G t , defined as the set of conic combinations of ∇ t L ∥∇L ⊥ r andEQUATIONfor two reasons. Firstly, all vectors in G t are easily computable due to the explicit expression of G t , whereas the dual cone K * is implicitly defined. Secondly, G t contains two important components of K * t , which are the projections of ∇L(θ t ) onto ∇L r (θ t ) ⊥ and ∇L b (θ t ) ⊥ by its construction. The next proposition shows that G t always belongs to the dual cone region as illustrated in Figure 3 . Proposition 4.3. Suppose that ∇L r (θ t ) and ∇L b (θ t ) are given at each iteration t. Consider G t , the set of conic combinations of ∇ t L ∥∇L ⊥ r and ∇ t L ∥∇L ⊥ b , defined in Eq. (4.1). Then, G t ⊆ K * t .Consequently, the DCGD algorithm defines the updated gradient denoted by g dual t within G t at each iteration t. A general framework for DCGD is presented in Algo 1.Require: learning rate λ, max epochT , initial point θ 0 for t = 1 to T do Choose g dual t ∈ G * t θ t = θ t-1 -λg dual t end forTo discuss the convergence properties of DCGD, we introduce the concept of Pareto optimality (adapted to the PINN setting), which is a key in multi-objective optimization [46, 40] . Definition 4.4. (Pareto optimal and stationary) A point θ ∈ R d is said to be Pareto-optimal if there does not existθ ′ ∈ R d such that L r (θ ′ ) ≤ L r (θ) and L b (θ ′ ) ≤ L b (θ).In addition, a point θ ∈ R d is said to be Pareto-stationary if there exists α 1 , α 2 such thatα 1 ∇L r (θ) + α 2 ∇L b (θ) = 0, α 1 , α 2 ≥ 0, α 1 + α 2 = 1.Intuitively, a Pareto-stationary point implies there is no feasible descent direction that would decrease all loss functions simultaneously. For example, consider a point θ t at which the cosine of the angle ϕ t between ∇L r (θ t ) and ∇L b (θ t ) is -1, i.e., cos(ϕ t ) = -1. Such a point is Pareto-stationary.The following theorem guarantees the convergence of the DCGD algorithm proposed in Algo 1 under some regularities in a non-convex setting. Assume L(θ * ) := inf θ∈R d L(θ) > -∞. Theorem 4.5. Assume that both loss functions, L b (•) and L r (•), are differentiable and the total gradient ∇L(•) is L-Lipschitz continuous with L > 0. If g dual t satisfies the following two conditions:(i) 2⟨∇L(θ t ), g dual t ⟩ -∥g dual t ∥ 2 ≥ 0, (ii) There exists M > 0 such that ∥g dual t ∥ ≥ M ∥∇L(θ t )∥, then, for λ ≤ 1 2L , DCGD in Algo. 1 converges to a Pareto-stationary point, or converges as Different variants of DCGD can be designed by properly choosing the updated gradient g dual t in G t satisfying the conditions (i), (ii) of Theorem 4.5. We present three specific algorithms: projection, average, and center.EQUATIONThe first algorithm, named DCGD (Projection), uses the projection of the total gradient ∇L(θ t ) onto G t when ∇L(θ t ) / ∈ K * t , which is the closest vector within G t to ∇L(θ t ). Specifically, the DCGD (Projection) algorithm specifies g dual t as follows: (i) ∇L(θ t ) if ∇L(θ t ) ∈ K * t , (ii) ∇ t L ∥∇L ⊥ r (c 1 = 1, c 2 = 0) if ∇L(θ t ) / ∈ K * t and ⟨∇L(θ t ), ∇L r (θ t )⟩ < 0, (iii) ∇ t L ∥∇L ⊥ b (c 1 = 0, c 2 = 1) if ∇L(θ t ) / ∈ K *EQUATIONwhich is geometrically interpreted as the projection of ∇L(θ t ) onto the angle bisector g c t of ∇L r (θ t ) and ∇L b (θ t ). The following proposition shows that g dual t of DCGD (Center) resides within G t Proposition 4.6. Consider the updated gradient g dual t of DCGD (Center) defined in Eq. (4.3). Then,g dual t ∈ G t .The visualization of these three algorithms can be found in Figure 4 and their pseudocodes are provided in Appendix E. Moreover, the proposed DCGD algorithms satisfy the conditions (i) and (ii) of Theorem 4.5. Consequently, the following Corollary summarizes the convergence of the proposed DCGD algorithms. Corollary 4.7. We impose the same assumptions as in Theorem 4.5. Then, DCGD (Projection), DCGD (Average), and DCGD (Center) converge to either a Pareto-stationary point or a stationary point.In addition to the theoretical result in Corollary 4.7, Appendix D.1 provides an ablation study on the empirical performance of three specific algorithms for solving benchmark PDEs. ) for each algorithm with φ max t = max{φ r t , φ b t } where φ r t is the angle between the updated vector and ∇L r (θ t ), and φ b t is the angle between the updated vector and ∇L b (θ t ).This subsection discusses benefits of DCGD through illustrative examples. We first investigate how the proposed DCGD algorithms resolve the conflicting gradient issue discussed in Section 3. Given each algorithm, at each iteration t, we define φ r t as the angle between the updated vector and ∇L r (θ t ), and φ b t as the angle between the updated vector and ∇L b (θ t ). Also, let φ max t = max{φ r t , φ b t }. We highlight that both φ r t and φ b t are less than π/2 under DCGD algorithms, as they ensure that the updated vectors always belong to the dual cone. Figure 5 plots the distributions of cos(φ max t ) for four different optimization algorithms: ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center) during the training of PINNs for solving the Helmholtz equation. It shows that three DCGD algorithms completely eliminate conflicting gradients in contrast to ADAM. Moreover, we observe that the distributions of cos(φ max t ) for DCGD (Projection) and DCGD (Average) are highly skewed toward zero, which implies that one of the two losses is unlikely to significantly improve. On the contrary, DCGD (Center) has a bell-shaped distribution with a mean of about 0.719, indicating that the two gradients are more aligned. This leads to a continuous reduction in both losses in a harmonious manner. Consistent with this observation, DCGD (Center) consistently outperforms DCGD (Projection) and DCGD (Average) in our experiments. Please refer to the ablation study D.1 for further comparisons. We empirically demonstrate that DCGD can converge to a Pareto-stationary point. Consider a (slightly modified) toy example shown in [37, 41] , which has two objective functions; see Appendix C.2 for more details. We solve the problem with 1,600 uniformly sampled initial points using ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center). Then, we mark with a red dot the point at which the algorithm fails to reach a Paretostationary point. Figure 6 shows that while ADAM does not reach a Pareto-stationary point across many areas, all DCGD algorithms achieve convergence to Paretostationary points throughout the entire space.Several MTL algorithms, such as PCGrad [37] , MGDA [40] , CAGrad [41] , Aligned-MTL [44] , and Nash-MTL [43] have been developed based on different and independent approaches. In contrast, the proposed DCGD framework provides a principled solution to the problem of conflicting gradients by directly characterizing the dual cone. As a result, our framework unifies many of these MTL algorithms as special cases, offering significant contributions not only to PINNs but also to the MTL domain. Proofs for the unification of MTL algorithms within the DCGD framework can be found in Appendix B.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] The pseudo code of our algorithms can be found in Appendix E, and we have stated detailed instructions in Appendix C. We have also provided code to reproduce the results of main experiments in supplementary material. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
numerical experiment: 
 This section demonstrates the superiority of DCGD through three distinct perspectives. In Section 5.1, we compare the performance of DCGD on five benchmark equations with that of a range of methods, including ADAM [45] , Learning Rate Annealing (LRA) [30] , Neural Tangent Kernel (NTK) [31] , PCGrad [37] , MGDA [40] , CAGrad [41] , Aligned-MTL [44] , MultiAdam [34] , and DPM [35] . Section 5.2 shows that DCGD can provide more accurate solutions for failure modes of PINNs and complex PDEs where vanilla PINNs fail. In Section 5.3, we explore the compatibility of DCGD with existing loss balancing schemes such as LRA and NTK.To compare the effectiveness of DCGD with other optimization algorithms, we measure the accuracy of the PINN solution trained by each optimizer using the relative L 2 -error. Then, we run each experiment across 10 independent trials and report the mean, standard deviation, max, and min of the best accuracy.We solve three popular benchmark equations (the Helmholtz equation, the viscous Burgers' equation, and the Klein-Gordon equation) and two high-dimensional PDEs (5D-Heat equation and 3D-Helmholtz equation) using vanilla PINNs with different optimization techniques. For DCGD, we employ an adaptive gradient version of the DCGD (Center) algorithm, the DCGD (Center) combined with ADAM (see Algo 5) by default for all experiments, provided in Appendix D.1. For other methods, we perform careful hyperparameter tuning based on the recommendations in their papers.The PDE equations and detailed experimental setting are provided in Appendix C.4. However, we do not report the performance of DPM because it is not only highly sensitive to hyperparameters but also exhibit poor performance, consistently observed in [47] .Table 1 : Average of relative L 2 errors in 10 independent trials for each algorithm on three benchmark PDEs (3 independent trials for two high-dimensional PDEs). The value within the parenthesis indicates the standard deviation. '-' denotes that the optimizer failed to converge. We explore more challenging problems, including failure modes of PINNs and complex PDEs, where vanilla PINNs fail to approximate solutions, and highlight the universal applicability of DCGD. We refer to Appendix C.5 for detailed experimental settings.First, we revisit the problem of a double pendulum in Steger et al. [32] , which is highly sensitive to initial conditions. The goal is to solve the trajectory of {(θ 1 (t), θ 2 (t))} t≥t0 , governed by the nonlinear differential equation as discussed in Eq. (C.1). The reference solution and its first-order derivative are represented by the blue solid and dotted lines, respectively, in Figure 7 . We train PINNs with SGD and ADAM to solve the double pendulum problem, where their solutions are depicted by the red solid and dotted lines in Figure 7a and Figure 7b , respectively. The PINN solutions trained with SGD and ADAM fail to accurately approximate the reference solution. In contrast, the reference solution is successfully recovered by our DCGD algorithm (see Figure 7c ). Second, we present the performance of DCGD for two challenging PDEs: the chaotic Kuramoto-Sivashinsky (KS) equation and the convection equation. For the chaotic KS equation, we combine DCGD with the causal training scheme of [22] , the current state-of-the art result. For the convection equation, DCGD is applied to PINNsFormer of [15] . As shown in Table 2 , DCGD achieves the lowest relative L 2 errors for the complex PDEs compared to the existing optimally tuned strategies, demonstrating its effectiveness in overcoming failure modes of PINNs. Third, the universal applicability of DCGD is not limited to specific architectures, sampling techniques, and training schemes. For example, A-PINN, designed for solving integral equations and integro-differential equations, achieves state-of-the art results in nonlinear Volterra IDEs [9] . DCGD significantly improves the performance of A-PINN for solving Volterra IDEs, as shown in Table 2 . Moreover, Table 4 shows that the performance of SPINN can be highly improved by applying DCGD for solving multi-dimensional PDEs.The proposed DCGD framework can be easily combined with existing PINN training strategies, including loss balancing methods. To illustrate this advantage, we have designed DCGD algorithms that integrate with LRA and NTK, named DCGD (Center) + LRA and DCGD (Center) + NTK, respectively. Please refer to Algo. 6 for the detailed implementation.We apply DCGD (Center) + LRA and DCGD (Center) + NTK to the same experiments described in Section 5.1. Tables 1 and 3 demonstrate that the performance of DCGD algorithms can be further enhanced across all the experiments in terms of the mean, maximum, and minimum of relative L 2 errors by integrating existing ideas from the literature.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] We have provided our code to reproduce the results of main experiments in supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and discussion: 
 In this work, we provided a clear criterion for when PINNs might be adversely trained, in terms of the angle and relative magnitude ratio of the gradients of the PDE residual and boundary loss, through a geometric analysis. Based on this theoretical insight, we characterized a dual cone region where both losses can decrease simultaneously without gradient pathologies. We then proposed a general framework for DCGD, which ensures that the updated gradient falls within the dual cone region, and provided a convergence analysis. Within this general framework, we introduced three specific DCGD algorithms and conduct extensive empirical experiments. Our experimental results demonstrate that the proposed DCGD algorithms outperform other optimization algorithms. In particular, DCGD is efficient in solving challenging problems such as failure modes of PINNs and complex PDEs compared to the current state-of-the art approaches. Furthermore, DCGD can be easily combined with other strategies and applied to variants of PINNs.Although we have presented a novel optimization algorithm, DCGD, to address challenging issues in PINNs, there still remain some interesting and important questions. For instance, one could design a more powerful DCGD specification within the dual cone region that goes beyond the projection, average, and center techniques. Also, while we mainly consider multi-objective optimization for PINNs, future work can focus on more general and complex types of multi-task learning problems.A Proofs for Section 4Proof of Theorem 4.2. Recall that ϕ t is the angle between ∇L r (θ t ) and ∇L b (θ t ), and R = ∥∇Lr(θt)∥ ∥∇L b (θt)∥ at each iteration t. Consider a cone K t , defined asK t := {cx|c ≥ 0, x ∈ {∇L r (θ t ), ∇L b (θ t )}} . Case (i). Suppose that ⟨∇L b (θ t ), ∇L r (θ t )⟩ ≥ 0. Observe that ⟨∇L(θ t ), ∇L r (θ t )⟩ = ⟨∇L b (θ t ), ∇L r (θ t )⟩ + ∥∇L r (θ t )∥ 2 ≥ 0, ⟨∇L(θ t ), ∇L b (θ t )⟩ = ⟨∇L b (θ t ), ∇L r (θ t )⟩ + ∥∇L b (θ t )∥ 2 ≥ 0.Therefore, by the definition of the dual cone, we have∇L(θ t ) ∈ K * t . Case (ii). Suppose that ⟨∇L b (θ t ), ∇L r (θ t )⟩ < 0 and -cos(ϕ t ) ≤ R ≤ -1 cos(ϕt) . By multiplying ∥∇L b ∥∥∇L r ∥ to -cos(ϕ t ) ≤ R, we get -cos(ϕ t ) ≤ R ⇔ -∥∇L b (θ t )∥∥∇L r (θ t )∥ cos(ϕ t ) ≤ ∥∇L r (θ t )∥ 2 , ⇔ ⟨∇L b (θ t ), ∇L r (θ t )⟩ + ⟨∇L r (θ t ), ∇L r (θ t )⟩ ≥ 0, ⇔ ⟨∇L(θ t ), ∇L r (θ t )⟩ ≥ 0. (A.1)On the other hand, by multiplying∥∇L b (θ t )∥ 2 cos(ϕ t ) to R ≤ -1 cos(ϕt) , we have R ≤ - 1 -cos(ϕ t ) ⇔ ∥∇L b (θ t )∥∥∇L r (θ t )∥ cos(ϕ t ) ≥ -∥∇L b (θ t )∥ 2 , ⇔ ⟨∇L b (θ t ), ∇L r (θ t )⟩ + ⟨∇L b (θ t ), ∇L b (θ t )⟩ ≥ 0, ⇔ ⟨∇L(θ t ), ∇L b (θ t )⟩ ≥ 0. (A.2) Therefore, we conclude that if ⟨∇L b (θ t ), ∇L r (θ t )⟩ < 0, then -cos(ϕ t ) ≤ R ≤ -1 cos(ϕt) is equivalent to ∇L(θ t ) ∈ K * t .Proof of Proposition 4.3. Recall thatG t := c 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b c 1 , c 2 ≥ 0 . (A.3)It is enough to show that we have ⟨g, ∇L r (θ t )⟩ ≥ 0 and ⟨g, ∇L b (θ t )⟩ ≥ 0 for any g ∈ G t . By the definition of G t , there existsc 1 , c 2 ≥ 0 such that g = c 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b for all g ∈ G t . One can easily check that ⟨g, ∇L r (θ t )⟩ = ⟨c 2 ∇ t L ∥∇L ⊥ b , ∇L r (θ t )⟩ = ⟨c 2 ∇L r (θ t ) - ⟨∇L b (θ t ), ∇L r (θ t )⟩ ∥∇L b (θ t )∥ 2 ∇L b (θ t ) , ∇L r (θ t )⟩ = c 2 ∥∇L r (θ t )∥ 2 - |⟨∇L b (θ t ), ∇L r (θ t )⟩| 2 ∥∇L b (θ t )∥ 2 = c 2 ∥∇L r (θ t )∥ 2 (1 -cos(ϕ t )) ≥ 0where ϕ t is the angle between ∇L r (θ t ) and ∇L b (θ t ). One can derive that ⟨g, ∇L b (θ t )⟩ ≥ 0 in the same manner. Therefore, we conclude that G t ⊂ K * t .Proof of Theorem 4.5. Let ϕ t be the angle between ∇L r (θ t ) and ∇L b (θ t ), and ψ t be the angle between g dual t and ∇L(θ t ) at the t-th iteration. Note that θ t+1 = θ t -λg dual t where g dual t satisfies the conditions (i), (ii) of Theorem 4.5.First of all, DCGD algorithm reaches a Pareto-stationary point if ϕ t = -1 by Definition 4.4, at which the optimization process is stopped.Otherwise, we first observe from the differentiability and L-Lipschitz continuity condition of ∇L(•) for all x, y ∈ R d :L(x) -L(y) = 1 0 ⟨∇L(y + t(x -y)), x -y⟩dt ≤ ⟨∇L(y), x -y⟩ + 1 0 ⟨∇L(y + t(x -y)) -∇L(y), x -y⟩dt ≤ ⟨∇L(y), x -y⟩ + 1 0 ∥∇L(y + t(x -y)) -∇L(y)∥∥x -y∥dt ≤ ⟨∇L(y), x -y⟩ + 1 0 Lt∥x -y∥ 2 dt = ⟨∇L(y), x -y⟩ + L 2 ∥x -y∥ 2 , (A.4)where we have used Cauchy-Schwarz inequality for the third inequality. Using Eq. (A.4) and Conditions (i), (ii) of Theorem 4.5, one calculates that for λ ≤ 1 2L ,EQUATION∥∇L(θ t )∥ 2 ∵ Cauchy-Swartz inequality and condition (ii) (A.5)By using telescoping sums, we further obtainT t=0 L(θ t+1 ) -L(θ t ) = L(θ T +1 ) -L(θ 0 ) ≤ - λM 2 T t=0 ∥∇L(θ t )∥ 2 ,which yields1 T + 1 T t=0 ∥∇L(θ t )∥ 2 ≤ 2 (L(θ 0 ) -L(θ T +1 )) λM (T + 1) ≤ 2 (L(θ 0 ) -L(θ * )) λM (T + 1) .Proof of Proposition 4.6. Note that g c t is the angle bisector of ∇L r (θ t ) and ∇L b (θ t ). From the formula of vector projection, g dual t of DCGD (Center) is the projection of ∇L(θ t ) on to g c t . Thus, it is enough to show that g c t is included in G t . We observe that∇ t L ∥∇L ⊥ r = ∇L(θ t ) -⟨∇L(θ t ), ∇L r (θ t )⟩ ∇L r (θ t ) ∥∇L r (θ t )∥ 2 (A.6) = ∇L b (θ t ) -⟨∇L b (θ t ), ∇L r (θ t )⟩ ∇L r (θ t ) ∥∇L r (θ t )∥ 2 , (A.7)and∇ t L ∥∇L ⊥ b = ∇L(θ t ) -⟨∇L(θ t ), ∇L b (θ t )⟩ ∇L b (θ t ) ∥∇L b (θ t )∥ 2 (A.8) = ∇L r (θ t ) -⟨∇L r (θ t ), ∇L b (θ t )⟩ ∇L b (θ t ) ∥∇L b (θ t )∥ 2 .(A.9)Then, by defining c 1 =1 ∥∇L b (θt)∥(1-cos(ϕt)) and c 2 = 1 ∥∇Lr(θt)∥(1-cos(ϕt)) , one can easily see thatc 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b = ∇L b (θ t ) c 1 -c 2 ⟨∇L r (θ t ), ∇L b (θ t )⟩ ∥∇L b (θ t )∥ 2 + ∇L r (θ t ) c 2 -c 1 ⟨∇L r (θ t ), ∇L b (θ t )⟩ ∥∇L r (θ t )∥ 2 = ∇L b (θ t ) ∥∇L b (θ t )∥ + ∇L r (θ t ) ∥∇L r (θ t )∥ = g c t . That is, g c t can be expressed as c 1 ∇ t L ∥∇L ⊥ r + c 2 ∇ t L ∥∇L ⊥ b for some c 1 , c 2 ≥ 0. Therefore, g c t is in G t .Proof of Corollary 4.7. We will show that g dual t of each DCGD algorithm satisfies the conditions (i), (ii) of Theorem 4.5. Three algorithms are summarized in Algo. 2, Algo. 3, and Algo. 4. We note that a conflict threshold α is introduced as a stopping condition for DCGD algorithms, as they can reach a Pareto-stationary point characterized by ϕ t = π. That is, the algorithm stops when the parameter converges close to a Pareto-stationary point such that π -α < ϕ t ≤ π. Here, we assume α ≥ 0 is fixed. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] We repeated the main experiments in several times and reported several statistics including min, max, and the standard deviations for our numerical results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] The overall experiments followed the existing experimental setup which have cited, and the code and data are available for use under the MIT license. For CausalPINNs in [22] , the code is available under the CC-BY-NC-SA 4.0 license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we prove that several MTL algorithms can be understood as special cases of the DCGD framework under the PINN's formulation.Proof. 1. MGAD [40] : The updated gradient g MGDA t of MGDA is defined by selecting the minimumnorm element from the convex combinations of ∇L r (θ t ) and ∇L b (θ t ) if there is gradient conflict ⟨∇L r (θ t ), ∇L b (θ t )⟩ < 0:g MGDA t := argmin α1,α2≥0 ∥u∥, s.t. u = α 1 ∇L r (θ t ) + α 2 ∇L b (θ t ), α 1 + α 2 = 1.One can easily show that ⟨g MGDA t , gMGDA t ⟩ = ⟨g MGDA t , ∇L r (θ t )⟩ = ⟨g MGDA t , ∇L b (θ t )⟩ ≥ 0. Thus, g MGDA t ∈ K * t .2. PCGrad [37] : PCGrad uses the same update direction with DCGD (Average) when ⟨∇L r (θ t ), ∇L b (θ t )⟩ < 0,g PCGrad t = 1 2 ∇ t L ∥∇L ⊥ r + ∇ t L ∥∇L ⊥ b ∈ K * t .and takes ∇L r (θ t ) + ∇L b (θ t ) when ⟨∇L r (θ t ), ∇L b (θ t )⟩ ≥ 0. The latter case is also contained in [43] : Nash-MTL considers a Nash bargaining solution to balance the loss gradients. The update gradient g Nash-MTL t is be defined byK * t . Therefore, g PCGrad t ∈ K * t . 3. Nash-MTLg Nash-MTL t := G t v t , (B.1) s.t. G ⊤ t G t v t = v -1 t . (B.2)whereG t = [∇L r (θ t ), ∇L b (θ t )].We can find the solution v t satisfying Eq. (B.2) as following. Byletting v t = v 1 v 2 , we have ∥∇L r (θ t )∥ 2 ⟨∇L r (θ t ), ∇L b (θ t )⟩ ⟨∇L r (θ t ), ∇L b (θ t )⟩ ∥∇L b (θ t )∥ 2 v 1 v 2 = 1 v1 1 v2, which is equivalent to∥∇L r (θ t )∥ 2 v 2 1 + ⟨∇L r (θ t ), ∇L b (θ t )⟩v 1 v 2 = 1 ∥∇L b (θ t )∥ 2 v 2 2 + ⟨∇L r (θ t ), ∇L b (θ t )⟩v 1 v 2 = 1 . (B.3) Therefore, we can derive v 2 = ∥∇Lr(θt)∥ ∥∇L b (θt)∥ v 1 . Substituting v 2 = ∥∇Lr(θt)∥ ∥∇L b (θt)∥ v 1 back into the first equation of Eq. (B.3) leads to ∥∇L r (θ t )∥ 2 v 2 1 + ∥∇L r (θ t )∥ 2 cos(ϕ t )v 2 1 = 1 ⇔ v 1 = 1 1 + cos ϕ t 1 ∥∇L r (θ t )∥ ⇔ v 2 = ∥∇L r (θ t )∥ ∥∇L b (θ t )∥ v 1 = 1 1 + cos ϕ t 1 ∥∇L b (θ t )∥ ⇔ G t v t = 1 1 + cos ϕ t ∇L r (θ t ) ∥∇L r (θ t )∥ + ∇L b (θ t ) ∥∇L b (θ t )∥where ϕ t is the angle between ∇L r (θ t ) and ∇L b (θ t ). Thus, the update gradient g Nash-MTL t has same direction with DCGD (Center). That is, g Nash-MTL t ∈ K * t .We conduct all experiments with PYTHON 3.10.9 and PYTORCH 1.13.1, CUDA 11.6.2, NVIDIA Driver 510.10 on Ubuntu 22.04.1 LTS server which equipped with AMD Ryzen Threadripper PRO 5975WX, NVIDIA A100 80GB and NVIDIA RTX A6000.We slightly modify the toy example in [41] to show our proposed method can expand the region of initial points that converge to the Pareto set. Consider the following loss functions with θ = (θ 1 , θ 2 ) ∈ R 2 :L 0 (θ) = L 1 (θ) + L 2 (θ) where L 1 (θ) = 2c 1 (θ)f 1 (θ) + c 2 (θ)g 1 (θ) and L 2 (θ) = c 1 (θ)f 2 (θ) + c 2 (θ)g 2 (θ), f 1 (θ) = log(max(0.5(-θ 1 -7) -tanh(-θ 2 ), 0.000005)) + 6, f 2 (θ) = log(max(0.5(-θ 1 + 3) -tanh(-θ 2 ) + 2, 0.000005)) + 6, g 1 (θ) = ((-θ 1 + 7) 2 + 0.1 • (-θ 2 -8) 2 )/10 -20, g 2 (θ) = ((-θ 1 -7) 2 + 0.1 • (-θ 2 -8) 2 )/10 -20, c 1 (θ) = max(tanh(0.5 • θ 2 ), 0), c 2 (θ) = max(tanh(-0.5 • θ 2 ), 0).The landscape and contour map of above loss function are shown in Figure 8 . The Pareto set is highlighted in gray in Figure 8b . We solve the above problem using ADAM, DCGD (Projection), DCGD (Average), DCGD (Center) for 100,000 epochs with different initial points. The initial points are selected as 1, 600 uniform grid points within [-10, 10] × [-10, 10]. Then, we mark with a red dot the point at which the optimizer fails to converge to the Pareto set.C.3 Details for Figure 2 , Figure 1 , and Figure 5 In this experiment, we use the 7-layer fully connected neural network with 20 neurons per layer and a hyperbolic tangent activation function. We train PINN models using SGD with the learning rate of 0.01 for 10, 000 epochs. In addition, 100 data points are sampled in boundaries and 10,000 points in the domain. The Helmholtz equation is described by∆u(x, y) + k 2 u(x, y) = f (x, y), (x, y) ∈ Ω, u(x, y) = 0, (x, y) ∈ ∂Ω, Ω = [-1, 1] × [-1, 1].The solution is given by u * (x, y) = sin(a 1 πx) sin(a 2 πy) wheref (x, y) = (k 2 -a 2 1 π 2 -a 2 2 π 2) sin(a 1 πx) sin(a 2 πy) In our experiment, we choose parameters: k = 1, a 1 = 1, a 2 = 4 as in [30] .The Viscous Burgers' equation is given byu t (t, x) + uu x (t, x) -νu xx (t, x) = 0, (x, t) ∈ [0, 1] × Ω, u(0, x) = -sin(πx), x ∈ Ω, u(t, -1) = u(t, 1) = 0, t ∈ [0, 1], Ω = [-1, 1]where ν = 0.01 π . The Klein-Gordon equation is∆u(t, x) + γu k (t, x) = f (t, x), (t, x) ∈ [0, T ] × Ω, u(0, x) = g 1 (x), x ∈ Ω u t (0, x) = g 2 (x), x ∈ Ω u(t, x) = h(t, x), (t, x) ∈ [0, T ] × ∂Ω Ω = [0, 1]We set parameters to k = 3, γ = 1, T = 1 and the initial conditions, g 1 (x) = x, g 2 (x) = 0 for all x ∈ Ω following [30] . Then we can use the solution u * (t, x) = x cos(5πt) + (tx) 3 where f (t, x) is derived by given equation .We employ a 3-layer fully connected neural network with 50 neurons per layer and use the hyperbolic tangent activation function for all experiments in Section 5.1. At each iteration, 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points. We just randomly sample the points in the boundaries if there exists an analytic solution, otherwise the points were resampled from a pre-generated set for each iteration. More specifically, for the case of Viscous Burger's equation, there is pre-determined 456 boundary points and we randomly sample in this set of points. We train PINNs for 50,000 epochs with Glorot normal initialization [48] using DCGD algorithms, ADAM [45] , LRA [30] , NTK [31] , PCGrad [37] , MultiAdam [34] , and DPM [35] .We search for the initial learning rate among λ = {10 -3 , 10 -4 , 10 -5 } and use a exponential decay scheduler with a decay rate of 0.9 and a decay step = 1, 000. For ADAM, we use the default parameters: β 1 = 0.9, β 2 = 0.999, ϵ = 10 -8 as in [45] . For LRA, we set α = 0.1, which is the best hyperparameter reported in [30] . For MultiAdam, we use β 1 , β 2 = 0.99 as recommended in [34] . For DPM, we test δ = {10 -1 , 10 -2 , 10 -3 }, ϵ = {10 -1 , 10 -2 , 10 -3 }, w = {1, 1.01, 1.001}.To compute the effectiveness of various optimization algorithms, we evaluate the accuracy of the PINN solutions u(•; θ) using the relative L 2 -error defined as:Relative L 2 error = N i=1 |u(x i ; θ) -u(x i )| 2 N i=1 |u(x i )| 2where u(•) is the true solution and {x i } N i=1 is the set of test samples. Unless the equation has an analytic solution, we use the numerical reference solution for u(x), which solved by finite element method [1] .In Table 3 , we report the best and worst-case relative L 2 errors of each method across 10 independent trials (3 independent trials for two high-dimensional PDEs). We plot the exact solution, PINN solution, and its error for each benchmark equation in Figures 10, 11 , and 9 The solution is given by u * (x, y) = sin(a 1 πx) sin(a 2 πy) sin(a 3 πz) wheref (x, y, z) = (k 2 -a 2 1 π 2 -a 2 2 π 2 -a 2 3 π 2 ) sin(a 1 πx) sin(a 2 πy) sin(a 3 πz) with k = 1, a 1 = 4, a 2 = 4, a 3 = 3.We employ a 5-layer fully connected neural network with 128 neurons per layer and use the hyperbolic tangent activation function. At each iteration, 128 points are randomly sampled in boundaries and 500 times more points in the domain as the collocation points. We train PINNs for 30,000 epochs with Glorot normal initialization.We use initial learning rate among λ = 10 -3 and use a exponential decay scheduler with a decay rate of 0.9 and a decay step = 1, 000.For 5-dimensional Heat equation, we follow the experiment setting in Hao et al. [49] . The PDE can be expressed as following:u t = k∆u + f (x, t), x ∈ Ω × [0, 1] n • ∇u = g(x, t), x ∈ ∂Ω × [0, 1] u(x, 0) = g(x, 0), x ∈ Ωwhere the geometric domain Ω = {x : ||x|| ≤ 1} and f (x, t) := - 1 d ||x|| 2 exp - 1 2 ||x|| 2 + t g(x, t) := exp - 1 2 ||x|| 2 + ty ′′ = f 1 (y, y ′ ) f 2 (y, y ′ ) subject to y(t 0 ) = θ 1 (t 0 ) θ 2 (t 0 ) , y ′ (t 0 ) = ω 1 (t 0 ) ω 2 (t 0 ) , (C.1)whereω 1 = θ1 , ω 2 = θ2 , f 1 (y, y ′′ ) = m 2 l 1 ω 2 1 sin(2∆θ) + 2m 2 l 2 ω 2 2 sin ∆θ + 2gm 2 cos θ 2 sin ∆θ + 2gm 1 sin θ 1 2l 1 (m 1 + m 2 sin 2 ∆θ) , f 2 (y, y ′′ ) = m 2 l 2 ω 2 2 sin(2∆θ) + 2(m 1 + m 2 )l 1 ω 2 1 sin ∆θ + 2g(m 1 + m 2 ) cos θ 1 sin ∆θ 2l 2 (m 1 + m 2 sin 2 ∆θ) .In this experiment, the initial conditions are θ1 (t 0 ) = θ 2 (t 0 ) = θ 0 = 150 • , ω 1 (t 0 ) = ω 2 (t 0 ) = 0.We replicate the experiment setup done in [32] . We use a six-layer feed-forward network of 30 neurons on each layer with the swish activation function. We train this model with ADAM optimizer with the default hyperparameters for 20,000 epochs. We also train the same model using DCGD (Center). Figure 13 shows the training curves of ADAM and DCGD. Convection equation We train PINNsFormer in Liu et al. [15] to solve convection equation which can expressed as following:u t + βu x = 0, ∀x ∈ [0, 2π], t ∈ [0, 1] u(x, 0) = sin(x),u(0, t) = u(2π, t) Where β = 50. we follow the default setting of [15] and train the model by 500 epochs. Auxiliary-PINN: Nonlinear integro-differential equation A-PINN is a variant of PINNs, designed to solve integro-differential equations [9] . We apply our DCGD algorithms to A-PINN for solving the following nonlinear 2-dimensional Volterra IDE:∂ 2 u(t, x) ∂t 2 = ∂u(t, x) ∂x - ∂u(t, x) ∂t -u(t, x) + g(t, x) + λ x 0 t 0 f cos(y 1 -y 2 )u(y 1 , y 2 )dy 1 dy 2where the boundary conditions are u(0, x) = x, ∂u(0,x) ∂t = sin(x), and u(t, 0) = t sin(t), 0 ≤ t, x ≤ 1. The analytic solution is u * (t, x) = x + t sin(t + x) with λ = 1 where g(t, x) is derived by given equation.Within the framework of A-PINN, it converts the above integral equation into the following equation by representing integrals as auxiliary output variables:∂ 2 u(t, x) ∂t 2 = ∂u(t, x) ∂x - ∂u(t, x) ∂t -u(t, x) + g(t, x) + λv(t, x), ∂v(t, x) ∂x = t 0 f cos(y 1 -x)u(y 1 , x)dy 1 = tw(t, x), ∂w(t, x) ∂t = cos(t -x)u(t, x),where the new variables v and w satisfies the boundary condition v(t, 0) = 0, w(0, x) = 0.For A-PINNs, we employ a 3-layer fully connected neural network with 50 neurons per layer and a hyperbolic tangent activation function. For training, 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points in each epochs. We train A-PINN models for 5, 000 epochs.Separable PINN: 3-dimensional Helmholtz equation SPINN is a a novel architecture designed to effectively reduce the computational cost of PINNs, especially when addressing high-dimensional PDEs [17] . To test the performance of DCGD for SPINNs, we consider the following 3-dimensional Helmholtz equation∆u(x, y, z) + k 2 u(x, y, z) = f (x, y, z), (x, y, z) ∈ Ω, u(x, y, z) = 0, (x, y, z) ∈ ∂Ω, Ω = [-1, 1] 3 .The solution is given by u * (x, y) = sin(a 1 πx) sin(a 2 πy) sin(a 3 πz) wheref (x, y, z) = (k 2 -a 2 1 π 2 -a 2 2 π 2 -a 2 3 π 2 ) sin(a 1 πx) sin(a 2 πy) sin(a 3 πz) with k = 1, a 1 = 4, a 2 = 4, a 3 = 3.We follow the optimal hyperparameter setting reported in [17] . For ADAM and DCGD (Center), the learning rate is 0.001. The input points are resampled every 100 epochs. Regarding model architecture, we use the SOTA model, so called (SPINN + Modified MLP). We record the mean and standard deviation of relative L 2 errors from 3 independent trials in Table 4 , indicating that the performance of SPINN can be significantly improved when trained with DCGD for a varying number of collocation points. D Supplemental resultsIn Section 4.3, we introduce three specific algorithms: DCGD (Projection), DCGD (Average), and DCGD (Center). We conduct an ablation study to investigate the impact of different updated gradient schemes within the dual cone region. More specifically, we compare the performance of these three algorithms on five benchmark equations. Detailed experimental settings can be found in Appendices C.4.Tables 5 and 6 demonstrate that DCGD (Center) outperforms the other DCGD algorithms across all experiments. Therefore, we consider DCGD (Center) as the default DCGD algorithm. In this section, we acknowledge that our proposed method incurs higher computational costs due to the need for backpropagation for each individual loss. Nonetheless, through a comparison of training speeds, we empirically demonstrate that DCGD achieves superior performance with computational costs comparable to those of existing competitors. Optimizerfor t = 1 to T do if π -α < ϕ t ≤ π or ∥∇L(θ t )∥ < ε then break end if if ∇L(θ t ) / ∈ K * then g dual t = 1 2 ∇ t L ∥∇L ⊥ r + 1 2 ∇ t L ∥∇L ⊥ b else g dual t = ∇L(θ t ) end if θ t = θ t-1 -λg dual t end for Algorithm 4 DCGD (Center) Require: learning rate λ, max epoch T , initial point θ 0 , gradient threshold ε, conflict threshold α for t = 1 to T do if π -α < ϕ t ≤ π or ∥∇L(θ t )∥ < ε then break end if g c t = ∇L b (θt) ∥∇L b (θt)∥ + ∇Lr(θt) ∥∇Lr(θt)∥ g dual t = ⟨g c t , ∇L(θt)⟩ ∥g c t ∥ 2 g c t θ t = θ t-1 -λg dualfor t = 1 to T do g dual t = DCGD(L r (θ), L b (θ)) g t ← g dual t m t ← β 1 m t-1 + (1 -β 1 )g t v t ← β 2 v t-1 + (1 -β 2 )g 2 t m t ← mt 1-β t 1 θ t ← θ t-1 -γ t mt √ vt+ϵend for Algorithm 6 DCGD with a loss balancing method Require: learning rate λ, max epoch T , loss balancing operator LB(•)for t = 1 to T do (β r , β b ) t = LB(L r (θ t ), L b (θ t )) L b (θ t ) ← β b L b (θ t ) L r (θ t ) ← β r L r (θ t ) Choose g dual t ∈ K * t θ t = θ t-1 -λg dual t end forNeurIPS Paper ChecklistIn practice, adaptive gradient descent algorithms such as ADAM[45] are widely employed.Our subsequent analysis in Section4.1 will clearly identify the extent to which significant differences in gradient magnitude lead to a challenge. 


Decision: This work introduces a novel approach for solving multi-objective optimization as arises in the training of physics-informed neural networks. The idea is simple and natural: to select an update direction that is guaranteed to be a descent direction for each component of the loss function individually. The authors achieve significant improvement in the resulting accuracy, on a wide range of PINN benchmarks. The approach is nonintrusive in terms of the other aspects of PINN training and might be equally useful for multiobjective training in other domains.

In summary, the method is well-motivated, has potential for broad applicability, and is thoroughly evaluated (at least in the domain of PINNs, which is the focus of this work) I therefore recommend acceptance.

Reviews: review: paper_summary: paper_summary: PINNs have become commonplace in both scientific computing and machine learning communities and have found widespread applications. There have been many modifications to various aspects of PINNs, such as the loss functions, initialisations, initial and boundary condition representations, learning algorithms and network architectures. 
A major problem with learning effective solutions through PINNs relative to other learning tasks is their difficulty in converging to the actual solution due to the complex nature of the underlying physics and equations involved and the corresponding loss landscape. 
The authors have provided a comprehensive survey of the previous works, making the unaddressed problems and goals very clear. A novel algorithmic framework based on convex optimization and geometric analysis is presented, which demonstrates superior performance on many tasks. 
The contributions of this paper are multifold, one being a new optimization algorithm while the other being a unified, open and flexible framework to adapt, modify and integrate the current technique with previous and future techniques.
The paper is well written with enough information on the theoretical and mathematical underpinnings, experimental setup, implementation details, results and justifications.
strength_weakness: strength_weakness: The authors have provided enough empirical and theoretical evidence for their proposed DCGD method, though since this work proposes a theoretical and algorithmic framework, albeit it would be notable to have a section on comparing the computational complexity and qualitative terms with existing algorithms. I would say a short section in Appendix D/E would go well.

Since the problem at hand is of multi-objective optimization which overlaps with multi-task learning problems and PINNs are a natural suitable choice, it would be very helpful to see the results of this approach on PINOs (Physics Informed Neural Operators). There is no comparison (even qualitative) of DCGD on PINOs (There are PINOs which are trained only on residual loss, but here I'm specifically referring to PINOs which include both residual and boundary loss for being relevant to DCGD). Was this not considered or omitted, or is it expected to show improvement in performance, as PINNs have shown? (Personally, I believe it should show improvements as the authors have already demonstrated superior performance on PINNformer with DCGD)


Presentations/Grammar/Typos (I do understand that the authors would be well aware of these and would rectify these in the pre-print or camera-ready versions. However, I would like to mention them here together to aid them. Some might have been missed; might update if needed)

Equation 2.2: The weight of the boundary loss is mentioned as $\omega_u$ and also on line 80, whereas it is mentioned as $\omega_b$ (which seems more appropriate) on line 79

Section 4.3: (This is a nitpick and can be conveniently ignored)
$g_t^c$ could be written on the same line as eq. 4.3 or as a newline as eq. 4.4 similar to how equation E.3 is written

Section 5.2: Is the title intended to be “P(I)DEs” or “PDEs”? I believe this is just a typo

Appendix A, line 769: Equations A.8 and A.9 could be mentioned here (as A.4 and A.5) and then referenced in the Proof of Proposition 4.6 section. I believe this will greatly enhance readability

Appendix A: Proof of Theorem 4.2 line 757: The residual loss is repeated twice $\nabla\mathcal{L}_r(\theta_t)$ is repeated twice, it should be both boundary and residual loss, $\nabla\mathcal{L}_r(\theta_t)$ and $\nabla\mathcal{L}_b(\theta_t)$

Appendix A, line770: "One can derive that …" should we include the boundary loss term instead of residual loss as the derivation with residual loss is already shown in line769 i.e it should be $\langle g, \nabla L_b (\theta_t) \rangle \geq 0$

Appendix A, line 781: For the sake of notational consistency, could you replace "condition (2)" with "condition (ii)"

Appendix A, line 785: (nitpick) The subscript t is omitted in the boundary loss and included in the residual loss ($\nabla\mathcal{L}_r(\theta_t)$ and $\nabla\mathcal{L}_b(\theta)$). I suggest the subscript "t" be included at both the places for notational consistency

Appendix A, line 818, Proof of Corollary 4.7:  3. DCGD (Center): “vector” instead of “veoctr”
questions: questions: I have gone through the previous works mentioned in detail, and does seems like the problem of multi-objective optimization or even optimization, specifically in the case of PINNs, has not been looked at from the perspective of Dual Cones. Could the authors add some more works related to Dual Cone-based techniques to the previous/related works section?

Typos and presentation issues are already covered in the weaknesses section; however, here, I would like to mention a few pointers where I would like to seek clarification. (This is more from a clarification perspective for me individually than a comment on the work, though I believe these clarifications will help the broader community)

Appendix A, line 779, Proof of Theorem 4.5: In the derivation of line779, from the first step to the next, should there be an equality sign in place of the first inequality sign as we are just distributing the dot product over addition on the first term $\nabla \mathcal{L}(y+t(x-y))$? (i.e. the first of the 3 inequality signs should be an equality instead)

Appendix A line 797: Is the expression $\left| \cos(\phi_t) - \pi \right| < \alpha$ is correct and as intended? The inequalities on lines 810 and 811 and the threshold criterion in the algorithms in Appendix E ($\pi - \alpha < \phi_t \leq \pi$) point otherwise. I believe there should not be the cosine of $\phi$; rather, it should just be the angle/phase difference $\phi$.
I have a similar comment for line 984, as the same expression is used.

Appendix C.5 line 925: Is the statement complete? Is it intentionally left this way?
limitations: limitations: I am appreciative of the authors who have mentioned (and also acknowledged in the checklist) a few limitations of the work in section 6 this is more from a future work perspective than from a limitations of the work presented as part of this paper. The limitations of the current work are not very evident from the present text.

It would be further helpful if they could include a more detailed section on where this cannot be applied or other multi-objective scenarios where the DCGD optimization framework is not as effective as other previous techniques.
paper_summary: paper_summary: In this paper, it is identified that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, the authors propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, the convergence properties of DCGD algorithms in a non-convex setting are analyzed. On a variety of benchmark equations, DCGD outperforms other optimization algorithms in terms of various evaluation metrics, achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs.
strength_weakness: strength_weakness: 1. It is better to provide the visualization of PINN predictions and compare with exact solutions, especially for the failure modes of PINNs.
2. Training timing is not reported.
questions: questions: Line 277, "Failure model" vs. "Failure modes"?
limitations: limitations: The proposed Dual Cone Gradient Descent (DCGD) optimization method for PINNs may still be trapped in local minima.
paper_summary: paper_summary: The paper concerns with the training of physics-informed neural networks (PINNs). It observes that in a multi-objective setting a naive gradient update which decreases the total objective might not decrease every individual objective. This is used to explain the challenge of training PINNs which have a potentially conflicting PDE and a boundary-condition loss. As a remedy, the work proposes to find another descent direction which decreases both of the PINN losses. Three different methods are proposed and the superior "center" method is selected for more detailed empirical investigation. It demonstrates strong results relative to several existing PINN optimizers.
strength_weakness: strength_weakness: However, the significance of the theoretical contributions must be questioned. There are just two theorems, of which 4.2 is fairly trivial and 4.5 is unclear: why is $\frac{1}{T+1}$ on both sides of Equation 4.2, and what does this equation even state? I cannot make sense of the provided intuition that the convergence rate is $\mathcal{O}(1/T)$, i.e., the more iterations, the slower you converge. Even more so, it is unclear how this intuition follows from Eq. 4.2 in the first place: it says that the sum of gradient norms is upper-bounded by the gap from the initial to the optimal loss, up to some constant.

The work also overlooks some recent literature explaining the difficult behavior of PINNs in terms of poor conditioning and proposing strong optimization strategies, which to the best of my knowledge achieve state-of-the-art convergence. I would argue these ideas should be included in the related work and as baselines contributing to a much more complete story.
- *De Ryck et al. An operator preconditioning perspective on training in physics-informed machine learning. ICLR 2024.*
- *Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.*

Several presentation choices could be improved, including minor mathematical and typing errors, odd wording, and superfluous parts (see suggestions below).

The work also overstates the practical success of PINNs: "a prominent approach for solving PDEs" (line 2), "remarkable empirical performance" (line 4), "PINNs have achieved great success in a wide range of applications" (line 8). This sets a misleading context for the practical significance of PINNs and by extension the presented work.
- *Grossmann et al. Can Physics-Informed Neural Networks beat the Finite Element Method? IMA Journal of Applied Mathematics 2024.*
questions: questions: - I would kindly ask the authors the clarify Theorem 4.5 in terms of the aforementioned concerns.
- How do the aforementioned optimization strategies compare to your experiments? Do you foresee a way for your method to be compatible to these (similar to section 5.3)?
- Does your method extend to more than two losses? Even in PINNs, there might be additional losses for data and initial conditions.
- How important is the assumption of fixing $\omega_r = \omega_b = 1$? Do the other optimizers show improved performance for other choices of these loss weights?
limitations: limitations: The limitations are discussed rather sparingly and should also include that the method currently applies to just two losses. It should also be noted that the performed experiments are closer to toy than practical problems. 

**Errors and suggestions**
- Line 59: It would be helpful if you briefly described what each of these three approaches do, instead of only listing examples.
- Line 73: $d$ is used for both the dimension of the domain as well as the parameter space.
- Eq. 2.2 and line 88: $\omega_u$ should be $\omega_b$
- Line 79: should this be an equality?
- I do not think writing out algorithm 1 adds much value as this is precisely gradient descent but with a different gradient.
- Line 170: should there be an argmin or arginf instead of inf? Or alternatively $\mathcal{L}(\theta*)$ instead of $\theta*$
- Lines 229-240 and Figure 6 might better belong to the appendix as it detracts attention from the main story.
- Line 185: "presents" -> "present"
- Line 221: "ADAM" -> "Adam"
- A.4: the first inequality should be an equality
- Adjust some odd wording, e.g. "mysterious challenges" (line 44), "harmoniously" (line 48)
- I suggest to moderate the claims about the practical success of PINNs (see weaknesses for concrete examples).
paper_summary: paper_summary: **Summary:** The paper proposes viewing the optimization of PINNs as a multi-objective optimization problem, with boundary and residual terms as potentially conflicting objective functions. To address conflicting gradients in the optimization process, the dual cone of the cone generated by the gradients of the boundary and residual terms is designated as the set of non-conflicting update directions. Several algorithms that guarantee updates from this dual cone are proposed, and their empirical performance is assessed.

**General Impression:** The manuscript is well-written, easy to follow, and generally of high quality. The numerical experiments are convincing and demonstrate the good performance of the proposed method. However, I disagree to a certain extent with the multi-objective viewpoint for the optimization of PINNs. For well-posed forward problems, the boundary loss and the interior loss are not really conflicting losses—at least if we assume that the network ansatz is sufficiently expressive. I expand more on this issue below in the questions. Nevertheless, the preprint presents a convincing suite of numerical experiments that seem to benefit from the multi-objective optimization approach.
strength_weakness: strength_weakness: See questions.
questions: questions: **Questions**

1. It is unclear whether PINN optimization should be viewed as a multi-objective optimization problem. Please comment.
2. I am missing a clarification on whether the proposed dual cone gradient descent is a completely novel idea, even in the field of multi-objective optimization, or if it is well-known in that field and its application to PINNs is novel. Please clarify and give references if appropriate.
3. Could the authors comment on how their proposed method relates to loss reweighting strategies? It seems that the proposed approach could equivalently be understood as a loss reweighting strategy.
4. Can the authors comment on the case of more than two loss functions.
5. Have the authors considered measuring angles in function space instead of parameter space by using the differential of the parametrization map? Recently, optimization methods that employ function space geometry have shown promise for the optimization of PINNs (see [1, 2, 3]). This might lead to further improvement of the proposed method.

**Details to question 1:**
A fundamental question for me is whether to view PINNs as a multi-objective optimization problem or not. To illustrate this question, consider the following academic example of a PDE with homogeneous forcing and inhomogeneous boundary data:

\begin{align}
Nu &= 0 \quad \text{in }\Omega,
\\
\newline
Bu &= g \quad \text{on }\partial\Omega.
\end{align}

Assume further that $N$ maps constant functions to zero (any PDE operator without a zeroth-order term). Assume a trial neural network is initialized representing (almost) the zero function (or any other constant function). Any change in the neural network will now likely increase the PDE residual (which is at its global minimum for any constant function). Furthermore, the increase in the PDE residual is necessary to obtain the correct solution. This might lead to a Pareto stationary point that has nothing to do with the sought-after solution.

**References**

[1] https://proceedings.mlr.press/v202/muller23b/muller23b.pdf

[2] https://openreview.net/pdf?id=z9SIj-IM7tn

[3] https://arxiv.org/pdf/2402.07318
limitations: limitations: See questions.

Comments: comment: Thank you for taking the time to review our paper and for raising our score. Below are our responses to your additional questions.

>**Concerning Q1**

Thank you once again for your deep and interesting question.

>**Connection to loss re-weighting strategies**

We agree with your observation. The proposed DCGD framework can be interpreted as a loss re-weighting strategy that ensures the prevention of gradient conflicts.

>**Function spaces**

For computing the natural gradients, we used the metric induced by the inner product in $H^1$ as in [1].

[1] Müller, J., & Zeinhofer, M. (2023, July). Achieving high accuracy with PINNs via energy natural gradient descent. In *International Conference on Machine Learning* (pp. 25471-25485). PMLR.
Thank you very much for your valuable comments and raising our score. 

Here are our responses to your additional questions.

- could you also report the performance of just DGCD on this task?
    
    The table below shows the performance of DCGD  without fine tuning with NNCG.
    
    | Optimizer | $L^2$ error |
    | --- | --- |
    | Adam | 2.12e-2 |
    | Adam + L-BFGS | 1.92e-2 |
    | DCGD | **1.20e-2** |
    
- I assume these can also be combined simultaneously by computing the gradients from NNCG for each loss and then combining them according to your method?
    
    There are two methods to combine DCGD with the straetegy of [1].
    
    **Combination 1**. Apply DCGD and NNCG sequentially
    
    First, train using DCGD for a certain number of epochs, and then fine-tune with NNCG (as similar to the approach of ADAM + L-BFGS + NNCG).
    
    **Combination 2**. Apply the updated gradients of DCGD to Each Optimizer
    
  For each optimizer—ADAM, L-BFGS, and NNCG—the updated gradients from DCGD are used, as described in the pseudo-code provided below.
      
    ```jsx
    adam_optimizer = Adam(model_params, adam_params)
    lbfgs_optimizer = L_BFGS(model_params, lbfgs_params)
    nncg_optimizer = NNCG(model_params, nncg_params)
    # for combined: adam_DCGD = True, lbfgs_DCGD = True, nncg_DCGD = True
    # for sequential (DCGD + NNCG): adam_DCGD = True, lbfgs_epoch = 0, nncg_DCGD = False
    adam_DCGD = True
    lbfgs_DCGD = True
    nncg_DCGD = True
    
    for i in range(adam_epoch):
    	if adam_DCGD == True:
    		# dcgd_closure set the param.grad as the dcgd update vector
    		adam_optimizer.step(dcgd_closure)
    	else:
    		# use loss.backward() where loss = loss_b + loss_r
    		adam_optimizer.step(closure)
    				
    for i in range(lbfgs_epoch):
    	if lbfgs_DCGD == True:
    		# dcgd_closure set the param.grad as the dcgd update vector
    		lbfgs_optimizer.step(dcgd_closure)
    	else:
    		# use loss.backward() where loss = loss_b + loss_r
    		lbfgs_optimizer.step(closure)
    		
    for i in range(nncg_epoch):
    	if nncgd_DCGD == True:
    		# dcgd_closure set the param.grad as the dcgd update vector
    		nncg_optimizer.step(dcgd_closure)
    	else:
    		# use loss.backward() where loss = loss_b + loss_r
    		nncg_optimizer.step(closure)
    ```
    

The results shown in Table 4 of the attached PDF were obtained using **Combination 2**. Additionally, we have also experimented with **Combination 1** and achieved further improvements. Due to time constraints, we conducted the experiments under the default parameter settings. Thus, there might be room for further improvement with hyperparameter tuning.

| Optimizer | $L^2$ error |
| --- | --- |
| Adam+L-BFGS+NNCG | 9.92e-3 |
| Combination 1 | **5.60e-3**|
| Combination 2 | 9.49e-3 |


[1] Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.
Thank you once again for reviewing our work. We are pleased that you are satisfied with our responses.
Thank the authors for their replies. I am satisfied with their answers and will maintain my score.
I kindly thank the authors for their insightful and strong rebuttal.

I was indeed not aware of this as a standard notation in the optimization literature. I appreciate you pointing this out, including the references, and I acknowledge my oversight on Theorem 4.5. I will also reduce my confidence rating on these grounds.

I also appreciate the theoretical and empirical comparison to the invoked references. For clarification, I referred to [4] achieving the lowest error, not the best convergence rate. It is impressive to see that DGCD can replace Adam + L-BFGS, when fine tuning with NNCG. If possible, could you also report the performance of just DGCD on this task?

I understand that this is combining the DGCD and NNCG sequentially. I assume these can also be combined simultaneously by computing the gradients from NNCG for each loss and then combining them according to your method?

Given that my main concerns have been addressed, as well as the discussion on the broader applicability beyond PINNs and extension to >2 losses, I will adjust my rating.
I thank you very much for the detailed answer. 

**Concerning Q1:** I understand your argument that the neural network can be trained in a way that does not increase the residual. I am still not fully convinced by the multi-objective viewpoint, but your experimental results are convincing and me not being convinced should not influence my assertion of your work. In any case, it is an interesting question.

**Connection to loss re-weighting strategies:** 
I believe it holds 
$$ 
G_t \subset \operatorname{span} ( \nabla \mathcal L_b, \nabla \mathcal L_r  )
$$
hence, no matter which update $g\in G_t$ is picked, we can write
$$
g = \alpha \nabla \mathcal L_b + \beta \nabla \mathcal L_r
$$
and hence view the update direction $g$ as stemming from the loss re-weighting
$$
\tilde{\mathcal L} = \alpha \mathcal L_b + \beta \mathcal L_r
$$
and $\nabla  \tilde{ \mathcal  L} = g$.

**Function spaces**
Thank you for the extra experiments. Could you elaborate which metric was chosen to compute the natural gradients?

I am convinced that this pre-print merits publication. I will raise my score.
Thank you for your feedback. We are glad that our rebuttals have addressed your concerns.
I have read through all the reviews, rebuttals and responses (including the general author rebuttal). I am satisfied with the authors' response and the general comment. I am pleased to see the applications of DCGD in unlearning problems. I would like to retain my review and the associated scores.