Dual Cone Gradient Descent for Training Physics-Informed Neural Networks
Authors: Youngsik Hwang
Dongyoung Lim
Venue: NeurIPS 2024 poster (2024)
Keywords: physics-informed neural networks;multi-objective optimization;scientific machine learning;gradient descent
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce a novel optimization algorithm, named dual cone gradient descent, for training PINNs.

Abstract: Physics-informed neural networks (PINNs) have emerged as a prominent approach for solving partial differential equations (PDEs) by minimizing a combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs often fail to generate reasonable solutions, and such pathological behaviors remain difficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, we propose a novel optimization framework, *Dual Cone Gradient Descent* (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in a non-convex setting. On a variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms of various evaluation metrics. In particular, DCGD achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD can be further improved by combining it with popular strategies for PINNs, including learning rate annealing and the Neural Tangent Kernel (NTK).

Dual Cone Gradient Descent for Training Physics-Informed Neural Networks
Youngsik Hwang Artificial Intelligence Graduate School UNIST hys38350unist..em kc
Dong-Young Lim"
Department of Industrial Engineering Artificial Intelligence Graduate School UNIST dl dlim@unist.c ac.kr
Abstract
Physics-informed neural networks (PINNs) have emerged a prominent approach for solving partial differential equations (PDEs) by minimizing combined loss function that incorporates both boundary loss and PDE residual loss. Despite their remarkable empirical performance in various scientific computing tasks, PINNs of- ten fail generate reasonable solutions, and such pathological behaviors remain dif- ficult to explain and resolve. In this paper, we identify that PINNs can be adversely trained when gradients of each loss function exhibit significant imbalance in their magnitudes and present negative inner product value. To address these issues, we propose novel framework for multi-objective optimization, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, we analyze the convergence properties of DCGD algorithms in non-convex setting. On variety of benchmark equations, we demonstrate that DCGD outperforms other optimization algorithms in terms various evaluation metrics. In particular, DCGD achieves superior predictive accu- racy and enhances the stability of training for failure modes of PINNs and complex PDEs, compared to existing optimally tuned models. Moreover, DCGD be further improved by combining it with popular strategies for PINNs, including learn- ing rate annealing and the Neural Tangent Kernel (NTK). Codes are available at https::/ //tpss://giian gig......... com/youngs welyyyunnnnkhhhe wkwwang/ Dual Cone Gradient-Descent
1 Introduction
Piysics-informed Neural Networks (PINNs) proposed Raissi et have created a new paradigm in deep learning for solving forward and inverse problems involving partial differential equations (PDEs). The key idea of PINNs is to integrate physical constraints, governed by PDEs, into the loss function of neural networks. This is in turn equivalent to finding optimal parameters for the neural network by minimizing a loss function that combines boundary loss and PDE residual loss. Thanks to their strong approximation ability and mesh-free advantage, PINNs have achieved great success in a wide range of applications
Building upon this success, the applications of PINNs have been extended to solve other functional equations, including integro-differentaal equations ( fractional PDEs [10] and stochastic PDEs [1I. Moreover, numerous variants of PINNs have been developed to enhance their computational efficiency and accuracy via domain decomposition methods 122 13 advanced neural network
architectures [14188, modified loss functions [1] different sampling strategies [22-24] and
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Recent studies have also explored optimizing PINNs by leveraging function space geometry, providing an alternative perspective to enhance accuracy and computational efficiency 27 28
probabilistic PINNs
25
26
Despite these achievements, several studies have reported that PINNs often fail learn correct solutions for given problems ranging from highly complex to relatively simple PDEs [29/31]. Due to the unclear nature of pathologies in the training of PINNs, has become a critical research topic to observed that PINNs tends to get stuck at trivial solutions while violating given PDE constraints over collocation points. The imbalance between PDE residual loss and boundary loss was explored in Wang et al. [30] and a spectral bias of PINNs was studied in Wang et al. [31]. Yao al. [34] discussed the gap between the loss function and the actual performance. Even with the insights from the aforementioned studies, a comprehensive understanding of PINN's failure modes remains largely unexplored in various scenarios.
explain and mitigate these phenomena. For example, 32 3
In this paper, we explore these challenges from a novel perspective of multi-objective optimization. We first provide a geometric analysis showing that PINNs can be adversely trained when the gradients of each loss function exhibit a significant imbalance in their magnitudes, coupled with a negative inner product value. Based on this finding, we characterize dual cone region where both PDE residual loss and boundary loss can be decreased simultaneously without the adverse training phenomenon. We then propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), for training PINNs which updates the gradient direction to be contained in the dual cone region at each iteration. Furthermore, we study the convergence properties of DCGD in a non-convex setting. In particular, we find that DCGD can converge to Pareto-stationary point. We validate the superior empirical performance and universal applicability of DCGD through extensive experiments.
2 Preliminaries
The Euclidean scalar product is denoted by (,, 1), with || standing for the Euclidean norm (where the dimension of the space may vary depending on the context). For subspace W of a vector space V, its orthogonal complement W is defined as
Notation.
W- == fv E Vi,, r) =0, 0, uE
For vector E V, the projection of on nontrivial subspace W is denoted by wpn. Unless otherwise specified, V represents Rdd throughout the paper.
Related Works. Among various research directions in PINNs, we focus on reviewing optimization strategies for PINNs. These can be broadly categorized into three main approaches: adaptive loss balancing, gradient manipulation, and Multi-Task Learning (MTL). As an example of adaptive loss balancing algorithms, Wang et al. (30 proposed learning rate annealing (LRA) algorithm that balances the loss terms by utilizing gradient statistics. Wang et al. (11 utilized the eigenvalues of the Neural Tangent Kernel (NTK) to address the disparity in convergence rates among different losses of PINNs. For gradient manipulation algorithms, the Dynamic Pulling Method (DPM) was proposed in (35 prioritize the reduction of the PDE residual loss. In [36], the authors used the PCGrad algorithm, proposed in (37], for training PINNs to address multi-task learning challenges. Li al. (38) developed an adaptive gradient descent algorithm (AGDA) that resolves the conflict by projecting boundary condition loss gradient to the normal plane of the PDE residual loss gradient. Yao al. [34] recently developed MultiAdam, a scale-invarian optimizer. to mitigate the domain scaling effect in PINNs. Another important line of gradient manipulation involves Multi- Task Learning (MTL) algorithms, which optimize single model to perform multiple tasks simultaneously 39 40 37 44 We will discuss that several MTL. algorithms can be unified within the proposed DCGD framework.
Informee Neural Networks. Let Ω C R be domain and U be the boundary of Ω. We consider the following nonlinear PDEs:
[u)((( = f(x), æ E ?
(2.1)
Blu](x) =((x,, an
where N and B denote a nonlinear differential operator and boundary condition operator, respec- tively. We approximate u(x) by a deep neural network u(æ; 0) parameterized by 0. To train the

---

neural network, the framework of PINNs minimizes the total loss function [(0), which is a weighted sum of PDE residual loss (0) and boundary condition loss ()))). defined by:
C(0): = w., wol with
(2.2)
1 N. Σ Wuu[= f(a,)1 Co(0) = No Σ (0) == N,
No
1
Blu[ g(x))13
where wy, w/b 2 0 are weights of each loss term, { denotes a set of collocation points that randomly sampled in Ω2 and { the boundary sample points. Here, we set wr = w/ 1
throughout the paper. We note that the training of PINNs falls into the category of multi-objective
learning due its structure of the loss function [(0) in Eq. 2.2
3 Empirical Observations and Issues in Training PINNS
This section investigates issues that are frequently ob- served during the training of PINNs in the context of multi-objectivve learning. The parameter for the PINN solution u( 0) is typically estimated by minimizing the total loss function [()) with (stochastic) gradient de- scent method
0.4
Co e VE K'
Loss
if V K'
0.1
0.0
500
1000
25:00
2300 Epoch
2000
5000
3500
4000
Be+1 = - -.)((),, tEN
Figure 1: Training curves for the total loss L (:= C, + Lx), PDE residual loss Cr, and boundary loss Ls for viscous Burgers' equation.
where VL(() is the gradient of the total loss function C(0) with respect to 6. However, a careless adoption of standard gradient descent methods may lead to an incorrect solution, as reducing the total loss does not necessarily imply a decrease in both the PDE residual loss and boundary loss. This phenomenon is clearly illustrated Figure [] which displays the curves of the total loss, PDE residual loss, and boundary loss over epochs for solving the viscous Burger's equation. Notably, while the total loss consistently decreases throughout the training, the PDE loss adversely increases.
Conflicting and dominating gradients in PINNs. This issue is highly related with discrepancies in the direction and magnitude between two gradients the PDE residual and boundary loss. Specifically, define two gradients to be confticting at the t-th iteration if they have negative x < Φi I π where o is the angle between (l, and V (6.)). When
inner product value, i.e., 2 there conflicting gradients, parameter updates minimize one loss function might increase the other, leading to an inefficient learning process such as oscillating between optimizing for the two loss functions and resulting in degraded solution quality 19]. Another problem arises when one gradient is much larger than the other, i.e., | (0,) & [VLa or IV, (0:) > (o The significant differences"] in the magnitudes of gradients in PINNs might create a situation where the optimization algorithm primarily minimizes one loss function while neglecting the other. This often results in slow convergence and overshooting, be more crucial in finding better solution. To mitigate the imbalance in the gradients, loss balancing approaches to rescale the weights of each loss term have been proposed 30
the smaller gradient, though neglected, may
To examine these challenges in training PINNs, we record cosine value the angle between VL, and VL., and the ratios of their magnitudes while training PINN for the Helmholtz equation. Figure shows that conflicting gradients are observed in about half of the total iterations. Moreover, we observe that the magnitude of the gradient of the PDE residual is several tens to hundreds of times larger than that of the boundary loss (See Figure b)). That is, conflicting and dominating gradients are prevalent issues the training of PINNS.
In practice, adaptive gradient descent algorithms such as ADAM 45 are widely employed. 'Our subsequent analysis in Section 4.1 will clearly identify the extent to which significant differences in gradient magnitude lead to challenge.
3

---

04
Densiyy
cass
-169 4.35 .43e -F25 c/e
61
COS
//VC//////sl
(a) histogram of (os
(b) histogram of R
Figure 2: Conflicting and dominating gradients in PINNs. Here, o is defined as the angle between V,, and VLs. R
|| is the magnitude ratio between gradients.
4 Methodology
In this section, we provide geometric analysis to identify a dual cone region where both PDE residual loss and the boundary loss can decrease simultaneously. Subsequently, introduce general framework for DCGD algorithms, ensuring that the updated gradient falls within this region. We then propose three specifications of DCGD algorithms: projection, average, and center. All proofs for main results in this section can be found in Appendix
4.1 Dual Cone Region
The concept of dual cone plays pivotal role in our DCGD algorithm. Formally, a dual cone is defined as a set of vectors that have nonnegative inner product values with a given cone.
Definition 4.1. (Dual cone) Let K be cone of Rd Then, the set
K
(ar, v) 2 for all a E K,y E
called the dual cone of K.
For each iteration I, consider a cone denoted by K,, which is generated by rays of two gradients, (0,) and (o.:)
K; = (cole 2 0, x E (V., (C.().)))
In the context PINNs, the dual cone of K,, denoted by K;, represents the set of gradient vectors where each vector is neither conflicting with the gradient of the PDE loss nor with the gradient of the boundary loss, i.c. for u E K;, (u. VC,(0,) 2 and (u, (L,) 2 0.
In other words, when the total gradient V ((,)) is in K; (as depicted by the region the red line in Figure T), the standard gradient descent taking the direction VL(,) will decrease both the PDE and boundary losses for a suitable step size. On the other hand, if V C(0,) & K; (the region indicated by the blue line in Figure D. one of the losses will adversely increase even with sufficiently small step sizes.
This indicates that the training process of PINNs significantly vary depending whether the total gradient belongs to the dual cone region. The following theorem establishes the necessary and sufficient conditions under which the total gradient falls within the dual cone region in terms of the angle and relative magnitude between the gradients of the PDE residual and boundary loss.
Theorem 4.2. Suppose that (.,) and (..(6:) are given at each iteration Let di be the angle between (O.) and (L,)),), and R [V2x(P.) ((.,) K; if and only if
(0)il be their relative magnitude. Then,
(1) (V,) (0.)) 2 0, or (ii (V), (O)) and cos 6: <
COS di
Theorem provides a clear criterion for when conflicting and dominating gradients lead to adverse training in PINNs. For instance, the condition (ii) in Theorem 4.2 implies that the larger d, (the more conflicting they are), even slight difference in their magnitudes can result in adverse training. In particular, Theorem 4.2 quantifies the extent of problematic relative magnitude between the two

---

K
G+
VLAAA V.L
V
VL(0)
(2,(%)
Figure 3: Visualization of dual cone region Ki and its subspace G2
gradients, thereby clarifying the concept of dominating gradients, which has not been previously defined in the literature.
Thus, our strategy aims to devise an algorithm that chooses the updated gradient within dual cone region at each gradient descent step. For notational simplicity, we write V.Livve and ViLivvg to represent V (0+) (V.C. and in simple and explicit subspace G2 defined as the of conic combinations of V.Live and Vivet
respectively. In particular, we are interested
G. : (ar V.LIiVE c2 ViC vV.. |,, 20)
(4.1)
for two reasons. Firstly, all vectors in G2 are easily computable due to the explicit expression of G whereas the dual cone K* is implicitly defined. Secondly, G, contains two important components of KÝ, which are the projections of V ((,)) onto (O,) and (L)) by its construction. The next proposition shows that G, always belongs to the dual cone region illustrated in Figure
Proposition 4.3. Suppose that V (O.) and V (Oc) are given at each iteration t. Consider Go the set of conic combinations of cfifee and V.LiVEs. defined in Eq. 14. Then, G, S K;.
Consequently, the DCGD algorithm defines the updated gradient denoted by gt within G, at each iteration I. A general framework for DCGD is presented in Algo T
Algorithm 1 Dual Cone Gradient Descent (base)
Require: learning rate X, max epoch T, initial point 0
for t 1 to 7 do
Choose dual E G 0, 0,-1 Agt dual
end for
4.2 Convergence Analysis
To discuss the convergence properties of DCGD, we introduce the concept of Pareto optimality (adapted the PINN setting), which key in multi-objective optimization 46 40
Definition 4.4. (Pareto optimal and stationary) A point E Rd is said to be Pareto-optimal if there does not exist 6 Rd such that
(0)) < (0) and (x( < (0))
In addition, a point& E Rd is said to be Pareto inntto-taationayy there exists a1 a2 such that (0) + a VL,(0) = O, 01.02 2 0,
+ a2 1.
Intuitively, Pareto-stationary point implies there is no feasible descent direction that would decrease all loss functions simultaneously. For example, consider a point 0, which the cosine the angle & between (O, and (L,( is -1, i.c., cos(ó,) -1. Such a point is Pareto-stationary,
5

---

The following theorem guarantees the convergence of the DCGD algorithm proposed in Algo some regularities in non-convex setting, Assume [(8*) := inf L(0) -o0.
Theorem 4.5, Assume that both loss functions, L-(-) and (r(-), are differentiable and the total
gradient V (.) is L-Lipschitz continuous with L > 0. If g satisfies the following two conditions:
(i) ((V)( (
[ 1 O,
(ii) There exists
such that ll (lse MIVV((0))
then, for X S 21, DCGD in Algo.
converges to Pareto-stationary point, converges
-
VC(OII < AM
(4.2)
T+1
Theorem 4.5 states that DCGD converges to either Pareto-stationary point, characterized by o, such that cos( = -1, or a stationary point at a rate of (L/VTT) in the nonconvex setting. Unlike single- objective (nonconvex) optimization where the goal is pursue stationary point, in multi-objective optimization, it ideal to find a Pareto-stationary point that balances all loss functions. Thus, DCGD offers significant theoretical and empirical advantages over popular optimization algorithms like SGD and ADAM, which are only guaranteed to converge to stationary point. The convergence of DCGD Pareto-stationary point empirically verified in Section 4.4
4.3 Dual Cone Gradient Descent: Projection, Average, and Center
Ge
G.
6
v(rl))))
VC.08)
(a) DCGD (Projection)
(b) DCGD (Average)
(c) DCGD (Center)
Figure 4: The updated gradient giun! of three DCGD algorithms.
Different variants of DCGD can be designed by properly choosing the updated gradient gلسوl in Ge satisfying the conditions (i), (ii) of Theorem 4.5 We present three specific algorithms: projection. average, and center.
The first algorithm, named DCGD (Projection), uses the projection of the total gradient VL(0.) onto G, when ((.,)) & K; which is the closest vector within G, to VL())) Specifically, the DCGD (Projection) algorithm specifies gftul follows: (i) VL(6.) if ((0,) K; (ii) V.Civet
(cr 1, 0) if (((,)) K; and (V.) (O.)) 0, (iii) V.Live (ci 0, 02 1) if VC((0)) & K; and (VL(O.), (O.) < O. See also Eq. and Algo
2
DCGD (Average) algorithm takes the average of V.L and V.Cives when the total gradient is outside K; i.e., 61 = 62 = if VL(6)) & K; See Eq. E.2 and Algo.
We note that both DCGD (Projection) and DCGD (Average) use VL(O,) as gi duall without any manipulation when (L(:)) E K; Moreover, they require determining if the total gradient is contained in the dual cone at each iteration, which may incur additional computational costs. On the other hand, gimal of DCGD (Center) is given by
(9f; VL(.))) |1af112
(L.)
VL, (C.II
dual
where of = (..)
(4.3)
which is geometrically interpreted as the projection of (L,)) onto the angle bisector of of VL, (0.) and (O..).) The following proposition shows that dual of DCGD (Center) resides within G,
Proposition 4.6. Consider the updated gradient golua of DCGD (Center) defined Eq. 4.3) Then, guaal E G

---

The visualization of these three algorithms can be found in Figure and their pseudocodes are provided Appendix E Moreover, the proposed DCGD algorithms satisfy the conditions (i) and (ii) of Theorem 4.5 Consequently, the following Corollary summarizes the convergence of the proposed DCGD algorithms.
Then, DCGD (Projection), DCGD (Average), and DCGD (Center) converge either a Pureto-stationary point or stationary point.
Corollary 4.7. We impose the same assumptions in Theorem 4.5
In addition the theoretical result in Corollary 4.7
provides ablation study on the empirical performance of three specific algorithms for solving benchmark PDEs.
Appendix D.1
casip
co
cos p max
CO5 p
(a) ADAM
(b) DCGD (Projection)
(c) DCGD (Average)
(d) DCGD (Center)
Figure 5: Distribution of cos(pp max) for each algorithm with pp max (Pf, + where yp the angle between the updated vector and (O:), and yot is the angle between the updated vector and (0.).
4.4 Benefits of the DCGD framework
This subsection discusses benefits of DCGD through illustrative examples. We first investigate how the proposed DCGD algorithms resolve the conflicting gradient issue discussed in Section 3 Given
each algorithm, at each iteration f, we define yo the angle between the updated vector and V C, (0z). and ob as the angle between the updated vector and V (6.). Also, let yp max (pt v We highlight that both qot and of are less than π 2 under DCGD algorithms, as they ensure that the updated vectors always belong to the dual cone. Figure 5 four different optimization algorithms: ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center) during the training PINNs for solving the Helmholtz equation. It shows that three DCGD algorithms completely eliminate conflicting gradients in contrast to ADAM. Moreover, we observe that the distributions of cos(y?mmr) for DCGD (Projection) and DCGD (Average) are highly skewed toward zero, which implies that one of the two losses is unlikely to significantly improve. On the contrary, DCGD (Center) has a bell-shaped distribution with mean of about 0.719, indicating that the two gradients are more aligned. This leads to continuous reduction in both losses in harmonious manner. Consistent with this observation, DCGD (Center) consistently outperforms DCGD (Projection) and DCGD (Average) in our experiments. Please refer to the ablation study D.] for further comparisons.
plots the distributions of cos cos(pppua for
We empirically demonstrate that DCGD can converge to a Pareto-stational point. Consider (slightly modified) toy example shown in [37 41. which has two objective functions; see Appendix C.2 for more details. We solve the problem with 1, 1,600 uniformly sampled initial points using ADAM, DCGD (Projection), DCGD (Average), and DCGD (Center). Then, we mark with a red dot the point at which the algorithm fails to reach a Pareto- stationary point. Figure that while ADAM does not reach a Pareto-stationary point across many areas, all DCGD algorithms achieve convergence to Pareto- stationary points throughout the entire space.
10
(a) ADAM
(b) DCGD
Figure 6: Toy example: the region where the algorithm fails to reach a Pareto- stationary point multi-objective opti- mization
Several MTL algorithms, such as PCGrad 37). MGDA 40 CAGrad [41) Aligned-MTL. [44, and
Nash-MTL 43 have been developed based on different and independent approaches, In contrast, the
proposed DCGD framework provides a principled solution problem conflicting gradients by directly characterizing the dual cone. As result, our framework unifies many of these MTL
7

---

algorithms as special cases, offering significant contributions not only to PINNs but also to the MTL domain. Proofs for the unification of MTL algorithms within the DCGD framework can be found in Appendix B
5 Numerical Experiment
This section demonstrates the superiority of DCGD through three distinet perspectives. In Section 5.1
we compare the performance of DCGD on five benchmark equations with that of range of methods,
including ADAM (45), Learning Rate Annealing (IRAA [30]. Neural Tangent Kernel (NTK) [31].
PCGrad (37], MGDA 40), CAGrad [41], Aligned-MTL [44), MultiAdam [34,, and DPM [35
Section 5.2 shows that DCGD can provide more accurate solutions for failure modes of PINNS and
complex PDEs where vanilla PINNs fail. In Section 5.3 we explore the compatibility of DCGD with
existing loss balancing schemes such as LRA and NTK.
To compare the effectiveness of DCGD with other optimization algorithms, we measure the accuracy of the PINN solution trained by each optimizer using the relative L' Then, we run each experiment across 10 independent trials and report the mean, standard deviation, max, and min of the best accuracy.
5.1 Comparison on benchmark equations
We solve three popular benchmark equations (the Helmholtz equation, the viscous Burgers' equa- tion, and the Klein-Gordon equation) and two high-dimensional PDEs (SD-Heat equation and 3D-Helmholl equation) using vanilla PINNs with different optimization techniques. For DCGD, we employ an adaptive gradient version the DCGD (Center) algorithm, the DCGD (Center) combined with ADAM (see Algo 55 by default for all experiments, provided in Appendix D.I. For other methods, we perform careful hyperparameter tuning based on the recommendations in their papers. The PDE equations and detailed experimental setting are provided in Appendix C.4 However, we do not report the performance of DPM because it is not only highly sensitive to hyperparameters but also exhibit poor performance, consistently observed in 47
Table 1: Average of relative L errors in 10 independent trials for each algorithm on three benchmark PDEs (3 independent trials for two high-dimensional PDEs). The value within the parenthesis indicates the standard deviation. '.' denotes that the optimizer failed to converge.
PDE equation
Optimizer
Helmholtz
Burgers'
Klein-Gordon
Heat (5D)
Helmholtz. (3D)
ADAM
0.0609 (0.0231)
0.0683 (0.0285)
0.0792 (0.0386)
0.0097 (0.0072)
0.6109 (0.2096)
LRA
0.0066 (0.0025)
0.0180 (0.0094)
0.0069 (0.0037)
0.0052 (0. (0.0056)
0.0831 (0.0123)
NTK
0.0358 (0.0107)
0.0224 (0.0061)
0.0223 (0.0151)
0.0027 (0.0012)
0.4037 (0.2620)
PCGrad
0.0109 (0.0031)
0.0159 (0.0061)
0.0286 (0.0064)
0.0083 (0.0049)
0.2532 (0.0476)
MGDA
0.7590 (0. 1180)
0.9780 (0.0462)
0.6690 (0.2790)
0.9883 (0.0217)
CAGrad
0.0735 (0.0390)
0.0321 (0.0063)
0.1850 (0.0301)
0.0043 (0.0016)
0.5854 (0.3032)
Aligned-MTL
0.6570 (0.0805)
0.0294 (0.0129)
0.5571 (8.1824)
0.0013 (0.0004)
0.9138 (0.0645)
MultiAdam
0.0211 (0.0032)
0.0875 (0.0303)
0.0228 (0.0038)
0.0009 (0.0007)
0.7809 (0.0031)
DCGD
0.0029 (0.0005)
0.0124 (0.0046)
0.0069 (0.0027)
0.0008 (0.0003)
0.0774 (0.0250)
DCGD+LRA
0.0023 (0.0007)
0.0104 (0.0021)
0.0050 (0.0013)
0.0012 (0.0005)
0.1045 (0.0485)
DCGDANTK
0.0057 (0.0035)
0.0113 (0.0040) 0.0055 (0.0014)
0.0009 (0.0004)
0.3525 (0.2659)
Table displays the mean and standard deviation of the relative 1 errors for each optimization algorithm applied the three PDE equations. The error plots of approximated PINN solutions and other statistics of relative L errors are summarized in Appendix In the result tables, we highlight the best and the second-best methods. While the second best methods vary across experiments, the proposed method consistently outperforms other algorithms achieving the lowest L errors. This result underscores the robustness and adaptability of our method for solving various PDEs.
5.2 Failure mode of PINNs and Complex P(DDEs
8

---

Aetranse Kebooe 5.
Adterac Appeeeo 6.ck
Aekorp Ko
MM k:
0.0 125 050
1.2
1.50
175 202
0.53
025
E52 0.5
1.75 200
021
250 0.25
2.00
(a) SGD
(b) ADAM
(c) DCGD
Figure 7: Double pendulum problem: prediction of each method. SGD and ADAM find shifted solutions, but DCGD successfully approximates the reference solution.
Table 2: Relative 1 errors for DCGD (Center) Chaotic KS equation, Convec- tion equation and Volterra IDEs.
We explore more challenging problems, including fail- ure modes of PINNs and complex PDEs, where vanilla PINNs fail to approximate solutions, and highlight the universal applicability of DCGD. We refer to Ap- pendix C.5 for detailed experimental settings.
Equation
Baseline
DCGD
Chaotic KS
0.0687
0.0376
First, we revisit the problem of a double pendulum in Steger et al. [32), which is highly sensitive to ini- tial conditions. The goal is to solve the trajectory of (t), (t)) governed by the nonlinear differential equation as discussed in Eq. C. The reference solution and its first-order derivative are represented by the blue solid and dotted lines, respectively, in Figure 7 We train PINNs with SGD and ADAM to solve the double pendulum problem, where their solutions are depicted by the red solid and dotted lines in Figure 77a and Fig- 7bl respectively. The PINN solutions trained with SGD and ADAM fail to accurately approximate the reference solution. In contrast, the reference solution is successfully recovered by our DCGD) algorithm (see Figure 7cc.. Second, present the performance of DCGD for two challenging PDFs: the chaotic Luramoto-Sivashinsky (KS) equation and the convection equation. For the chaotic KS equation,
Convection
0.4880
0.0246
Volterra IDEs
0.0068
0,0011
combine DCGD with the causal training scheme of [22]. the current state-of-the art result. For the convection equation, DCGD is applied to PINNsFormer of [15]. As shown in Table 2, DCGD achieves the lowest relative L errors for the complex PDEs compared to the existing optimally tuned strategies, demonstrating its effectiveness in overcoming failure modes of PINNs. Third, the universal applicability of DCGD is not limited to specific architectures, sampling techniques, and training schemes. For example, A-PINN, designed for solving integral equations and integro-different equations, achieves state-of of art results nonlinear Volterra IDEs (9. DCGD significantly improves the performance of A-PINN for solving Volterra IDEs, as shown in Table 2 Moreover, Table shows that the performance of SPINN can be highly improved by applying DCGD for solving multi-dimensional PDEs.
5.3 Compatibility of DCGD with existing methods
The proposed DCGD framework can be easily combined with existing PINN training strategies, including loss balancing methods. To illustrate this advantage, we have designed DCGD algorithms that integrate with LRA and NTK, named DCGD (Center) LRA and DCGD (Center) NTK, respectively. Please refer to Algo. for the detailed implementation.
We upply DCGD (Center) LRA and DCGD (Center) + NTK to the same experiments described in Section 5. Tables [ and demonstrate that the performance of DCGD algorithms can be further enhanced across all the experiments in terms of the mean, maximum, and minimum of relative 7 errors by integrating existing ideas from the literature.
Conclusion and Discussion
In this work, we provided a clear criterion for when PINNs might be adversely trained, terms the angle and relative magnitude ratio of the gradients of the PDE residual and boundary loss, through a geometric analysis. Based on this theoretical insight, we characterized dual cone region where both losses can decrease simultancously without gradient pathologies. We then proposed general

---

framework for DCGD, which ensures that the updated gradient falls within the dual cone region, and provided a convergence analysis. Within this general framework, we introduced three specific DCGD algorithms and conduct extensive empirical experiments. Our experimental results demonstrate that the proposed DCGD algorithms outperform other optimization algorithms. is efficient in solving challenging problems such failure modes of PINNs and complex PDEs compared to the current state-of-the art approaches. Furthermore, DCGD can be easily combined with other strategies and applied to variants of PINNs.
particular, DCGD)
Although we have presented a novel optimization algorithm, DCGD, address challenging issues in PINNs, there still remain some interesting and important questions. For instance, one could design more powerful DCGD specification within the dual cone region that goes beyond the projection, average, and center techniques. Also, while we mainly consider multi-objective optimization for PINNs, future work can focus on more general and complex types of multi-task learning problems.
Acknowledgement
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) R0SS-202-000220000 the Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2020) 0-01336, Artificial Intelligence Graduate School Program (UNIST)), and Startup Research Fund (1.220132.01) of UNIST (Ulsan National Institute of Science & Technology).
References
[1] Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physies-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlincar partial differential equations. Journal of Computational physics. 378:686-707, 2019.
[2] Erik Laurin Strelow, Alf Gerisch, Jens Lang, and Mare E. Pfetsch. Physics informed neural networks: A case study for gas transport problems. Journal of Computational Physics, 481:112041, 2023. ISSN 0021-9991.
[3] Pushan Sharma, Wai Tong Chung, Bassem Akoush, and Matthias Ihme, A Review of Physics- Informed Machine Learning in Fluid Mechanics. Energies, 16(5):2343, 2023.
[4] Peter R. Wiecha, Arnaud Arbouct, Christian Girard, and Otto Muskens. Deep learning in nano-photonics: inverse design and beyond. Photon. Rex., 9(5):B182-BBB May 2021.
[5] Mahmudul Islam, Md Shajedul Hoque Thakur, Satyajit Mojumder, and Mohammad Nasim Hasan. Ex- traction of material properties through Computational Materials Science, 188: 1 1011111111000 2021.
deep learning from molecular dynamics simulation.
[6] Jonthan D Smith, Zachary E Ross, Kamyar Azizzadenesheli, and Jack B Muir. HypoSVI: Hypocentre inversion with Stein variational inference and physics informed neural networks. Geophysical Journal International, 228(1 ()68--710, 2022.
[7] Yogesh Verma, Markus Heinonen, and Vikas Garg, ClimODE: Climate Forccasting With Physics-informet Neural ODEs. In The Twelfth International Conference on Learning Representations, 2024.
[8] Yuyan Ni, Shikun Feng, Wei- Yeng Ma. Zhi-Ming Ma. and Yanyan Lan. Sliced Denoising: A Physics- Informed Molecular Pre-Training Method. arXiv preprint urXiv:2311.02122 2023.
[9] Lei Yuan, Yi- Yi-Qing Ni, Xiang- Yun Deng, and Shuo Hao. A-PINN: Auxiliary physies informed neu- ral networks for forward and inverse problems of nonlinear integro diffgro-diffcrentiall equations. Journal of Computational Physics, 462: 122:11220 112600 2022.
[10] Guofei Pang, Lu Lu, and George Em Karniadakis. fPINNs: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4):22003A2226 2019.
[11] Dongkun Zhang, Ling Guo, and George Em Karniadakis. Learning in modal space: Solving time-dependent stochastic PDEs using physics-infonin informeed neural networks. SIAM Journal on Scientific Computing, 42(2): A369-A665, 2020.
[12] Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. hp- VPINNs: Variational physics- informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering, 374:113547, 2021.
10

---

[13] Ameya D Jagtap and George E Karniadakis. Extended Physics-ine -nysome- Neural Networks (XPINNs): A Generalized Space-Time Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations. AAAI spring symposium: MLPS, volume 10, 2021.
[14] Benjamin Wu, Oliver Hennigh, Jan Kautz, Sanjay Choudhry, and Wonmin Byeon. Physies Informed RNN-DCT Networks for Time- -eme-Depnddent Partial Differential Equations. In International Conference on Computational Science, pages 372-379 Springer, 2022.
[15] Liyang Liu, Yi Li, Zhanghui Kuang, J Xue. Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. PINNsFormer: A Transformer-Base Based Framework For Physics-Informed Neural Networks. International Conference on Learning Representations. 2024.
[16] Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork-base Meta-Learning for Low-Rank Physics-Informed Informed Neural Networks. In Advances in Neural Information Processing Systems, 2023.
[17] Junwoo Cho. Seungtae Nam, Hyunmo Yang, Seok-Bae Yun, Youngjoon Hong, and Eunbyung Park. Separable Inffomee- Neural Networks. In Advances in Neural Information Processing Systems, 2023.
[18] Jihun Han and Yoonsang Lee. Hierarchical learning to solve partial differential equations using physics- informed neural networks. arXiv preprint wxXiv2221.0000444 2023.
[19] Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Kariiadakis. neural networks for forward and inverse PDE problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823, 2022.
physics-informet
[20] Hwijae Son, Sung Woong Cho, and Hyung Ju Hwang. Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNss). Neurocomputing, page 126424, 2023.
[21] Chuwei Wang, Shanda Li, Di Hc, and Liwei Wang. Is L,2 Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? In Advances in Neural Information Processing Systems, volume 35, pages 8278-8290, 2022.
[22] Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404 2022.
[23] Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. comprehensive study of non-adaptive and sasiduallbased adaptive sampling for physics-informe neural networks. Computer Methods in Applied Mechanics and Engineering, 403:1 1 15671, 2023.
[24] Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Mitigating Propagation Failures in Physics-informed Neural Networks using Eetain-Ressamle--ee Release (R3) Sampling. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7264-7302. PMLR, 23-29 Jul 2023.
[25] Amaud Vadeboncocur, Ömer Deniz Akyildiz, Ieva Kazlauskaite, Mark Girolami, and Fohmi Cirak. Fully probabilistic deep models for forward and inverse problems in parametric PDEs. Journal of Computational Physics, 123:9, 2023.
[26] Arnaud Vadeboncoeur, Teva Kazlauskaite, Yanni Papandreou, Fehmi Cirak, Mark Girolami, and Omer Deniz Akyildiz. Random Grid Neural Processes for Parametric Partial Differential Equations. In Proceedings of the 40th International Conference on Machine Learning, volume 202 Proceedings of Machine Learning Research, pages 94759-34778, 23-29 Jul 2023.
(27] Johannes Müller and Marius Zeinhofer. Achieving High Accuracy with PINNs via Energy Natural Gradient Descent. In International Conference Machine Learning, pages 25471 25471-25488 1-24855. PMLR, 2023.
[28] Johannes Müller and Marius Zeinhofer. Position: Optimization in SciMI. Should Employ the Function Space Geometry. In Forty-furst International Conference Machine Learning, 2024.
[29] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics- informeed neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 26548-26560, 2021.
[30] Sifan Wang. Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informe informed neural networks. SIAM Journal Scientific Computing, 43(55: 2021.
11

---

[31] Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physicx, 449:1 2022.
[32] Sophie Steger, Franz M. Rohrhofer, and Bernhard C. Geiger. How PINNs cheat: Predicting chaotic motion of double pendulum. In The Symbiosis of Deep Learning and Differential Equations II, 2022.
[33] Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.
[34] Jinchen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of inforred Neural Networks. In International Conference on Machine Learning, volume 202 of Proceedings of Muchine Learning Research, pages 39702-39721. PMLR, 23-29 Jul 2023.
[35] Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin, and Noseong Park. DPM: ing method for physics-informe neural networks in extrapolation. In AAAI Conference on Artificial Intelligence, number 9, pages 8146 8154, 2021.
A novel train-
[36] Bahador Bahmani and WaiChing Sun. Training multi-abjeettivemmlittaaa collocation physics-informed neural network with student/teachers transfer learnings. arXiv preprint arXiv:2107. 1at.:20077.1196,, 11996, 2021.
[37] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Task Learning. In Advances in Neural Information Processing Systems, volume 33, pages
2020.
[38] Xiaojian Li, Yuhao Liu, and Zhengxian Liu. Physics infiicce-ifomeedd neural network based on new adaptive gradient descent algorithm for solving partial differential equations of flow problems. Physics of Fluids, 35(6), 2023.
[39] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization, 2018
[40] Jean-Antoine Désidéri. Multppeegraddenn descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350(5)3313-318 2012.
[41] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-Averse Gradient Descent for Multi- task learning. In Advances in Neural Information Processing Systems, volume 34, pages 18878 18890. 2021.
[42] Liyang Liu, Yi Li, Zhanghui Kuang, J Xue. Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. International Conference on Learning Representations, 2021.
[43] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron. Kenji Kawaguchi. Gal Chechik, and Ethan Fetaya. Multi-Task Learning as a Bargaining Game In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16428-166 16446. PMLR, 17-23 Jul 2022.
[44] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent Component Aligument for Multi- Task Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20083-20093, 2023.
[45] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6800 (6800, 2014.
[46] Harold M. Hochman and James D. Rodgers. Pareto optimal redistribution. The American economic review, 5944)5542-557 1969.
[47] Lukas Fesser, Richard Qiu, and Luca D' "'immmoo-oor Wong. Understanding and Mitigating Extrapolation Failures in Physics-Ifformed Informed Neural Networks. arXiv preprint wrXiv:2306.09478.2 2023.
[48] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statisties, pages 249-256, JMLR Workshop and Conference Proceedings, 2010.
[49] Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su. Ziao Wang, Fanzhi Lu. Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, al. PINNacle; A comprehensive benchmark of physics-informed neural networks for solving PDEs. arXiv preprint 2882777,00. 2023.
12

---

A Proofs for Section
Proof of Theorem 4.2 Recall that Φr is the angle between V2,(0,) and VL,(0,), and
IV (0,)ll at each iteration Consider a cone K₁ defined as
(P))
K, = (crlc 0, E (V., (L.()))
Case (i). Suppose that (VL)(0)), (.)( 2 0. Observe that (VL())), (o.) = (V.) VL. + (0.) 20, (VL())), (L.)).)) = (VLs(0)), VL. + (0,)) O. Therefore, by the definition of the dual cone, we have V(((O)) E K
Case (ii). Suppose that (VL)(O)), (O,( < 0 and - cos( S R S [00 By multiplying
I/ VI, cos( cos(Ør) R, get
cos( R
VV. |I|| [V
- (0.)3.
(L(,)) VL,( + (VL, VL.( 2 0,
(VC(8),, (C,( 20. 0.
(A.1)
On the other hand, by multiplying VCC((:)]] cos( R < cos( we have
S cas(
([C.(.))II cos( -
(LC)(0.). vC,( + (V,, VL,(0.)) 2 0, (V))))) vL,(O:)) 20.
(A.2)
Therefore, we conclude that if (VL,(0,), vL,(0.)) < 0, then - cos(ó.) S R < ( is
equivalent to V.((O.) E K;
Proof of Proposition
4.3
Recall that
G: = {a faVV + e2 V.L ve |1, 2 20)
(A.3)
enough to show that we have (g. (O.)) 2 0 and (g. vL2(0.)) 2 for any g E G, By the definition of G, there exists cı, 02 such that g ci V.CVEe+ Vefifet for all g E G. One can easily check that
(g. V (o)))) = (c2 ViLive, VC,(
(O,) (VC)(0)), (O.)
VCO(O)) VC.(
(02
(0,))1
(IVC, (8.)11 (O.), vC,
c2
[V
02 (2,(00)) cos(
20 0
where de is the angle between (C, and (L).))). One can derive that (g. vLz(O)) 2 in the same manner. Therefore, conclude that G2 K
Proof of Theorem 4.5 Let de be the angle between (O.) and (Ls and wh, be the angle between ge dual and (Z( at the 1-th iteration. Note that 0c+1 O, >gg. dual where gf conditions (i), (ii of Theorem 4.5
dual satisfies the
13

---

First of all, DCGD algorithm reaches a Pareto-stationar point if o, = -1 by Definition 4.4 at which the optimization process stopped.
Otherwise, we first observe from the differentiability and I.-Lipschitz continuity condition of VZ(-) for all z, y E R
((x) L(v) S' (VL(y + 1)), x --yydd
I (VC(y), I 5-9) 9) + / (VC(y + 1(x--1) Vc(y), y)dt
s (VL(y), x - v) + [ |V 1(x - --)) v - y/kd
< (VL(y), x- - 9) + / Le|la y de
(VC(y), x y) + |1 y|1?
(A.4)
where we have used Cauchy- Schwarz inequality for the third inequality. Using Eq. (A.4) and Conditions (i), (ii) of Theorem 4.5 one calculates that for À 27.'
LA²
[(0++1) - [(O,) v -)())(0)), dual
+
197
dual ||
2
S A(V((())) dual
+ -
119 dual ||
(2(VL(()),
gt
llgt dual ||2 + +2(VL((),, gt duai))
S (VL())) dualy
condition
AM
(V((2))
Mauchy-Swarth inequality and condition (ii) (A.5)
2
By using telescoping sums, further obtain
T
Σ ((++1) -((,)) = [(0r+1)- - L(bv)
AM
$ 2
VL(())1
which yields
1
2 (C((0) [(0r+1))
VC(00)
T t=
AT 1)
2 (L((0) - L(O7)) AM(T +1)
Proof of Proposition 4.6 Note that of is the angle bisector of (L,(0:) and (L,) From the formula of vector projection, ادسوو of DCGD (Center) is the projection of V ((e,) on to gf Thus, it is enough to show that of is included in G,,
We observe that
(L,
VeL VL(O.) (VC())), (o.)
(A.6)
IVC, (0:)12
VC,(O,)
vc.( (VL)()), VL)(
(A.7)
[C.)())
14

---

and
(C)(
V.LivvE = VL(O.) - (VL(O)), v..(O.))
(A.8)
V La (0:)112
(L+(
(..) (O), VC.(0.))
(A.9)
VLL(0,)]1
Then, by defining 01 [V2x(0))111 (0)
and 02
[V2, cx, one easily see that
(VC, (0.), VC)(
a ViLLIEL + VVV.LiiV (L,(0.)
ci -1 02
I (V
(VL, (0:), (..(
+ (O.)
02 --- c
IVVL((()))
(LL(
+ (O+(
(())l
(
-6-
That is, of can be expressed ci V.Live + 02 VeLive for some C1 C2 2 Therefore, 9f is in G,,
Proof of Corollary 4.7 We will show that gسسal of each DCGD algorithm satisfies the conditions (i),
(ii) Thehree 4.5 Three algorithms are summarized in Algo. 2 Algo. 3 and Algo. 4 We note that conflict threshold is introduced as stopping condition for DCGD algorithms, as they can reach -trtto-stationary point characterized by Φt = л. That is, the algorithm stops when the parameter Here, we assume 0 is
converges close to a Pareto-stationary point such that a < 0i fixed.
1. DCGD (Projection): Note that it is trival to show that g VL((,), when (V)())) (.,
0, satisfies the conditions (i), (ii). Thus, we focus on the case when (V,) (0.)) 0.
First of all, need to show the condition (i)
2 (VV)(,)), St dual) l|st qual [VV((0)) 11 duan) VL(0.) 2 0,
which is equivalent that VC(0.)II للس V VL()))) Using Eq. A.6).. one directly calculates that
(O.) VVC, (0,)
VC(O)) (VC())). VC,(
In the same manner, we have 122 - V = [1 Since اسسدو is chosen in G,,
specifically 6 = 1,02 = 0 or = 0, 0202 O. we can write
_ V2(0,)11
9 VC (8)) + + VC (Bi) VC((.)
41 VC (0)) + 02 VC (0:))
C1 + 02 (Z(0.)
(1
(1- (att02 -1)
2 -1) ////o))
<
C
(2V4 (0.) -V(())
/|
+
C2 2V L (0.) - -VC(0)))))))))) | (1(
+ - 1) vc(0))
2
2
c [V
2 Vi(
c1 + 02
|C()))
2
+
[ + 02 -1) 1)) |V
+
(A.10)
2
[[))))
where we have used C + C2 for obtaining the last inequality. Therefore, the condition (i) is satisfied.
15

---

We further suppose that (V2(0)), vC.(O:)) < O. Then, gtti V.Liiet Let de be the angle between (O:) and (0,, and dh be the angle between gi and VL(O.). Note that de S n-a where a is conflict threshold. Otherwise, the algorithm stops when < de S (see Algo. 2
Then, since 1h+ oi 2
R we have
119 dual = VV ic lI
V (()))) cOs(14))
[VC(())) cos (b
||V[()))
Thus, by choosing M cos ( a). the condition (ii) satisfied. We repeat the same analysis for the case when (VL(0,), vC)(O.))
DCGD (Average): Similarly to DCGD (Projection), we focus on the case where DCGD (Average) specifies c = C2 = 2 given by
2 ViLLIVC + veC(fot))
when VC((0)) & K; Eq A.10 with 0
c2 1/2 directly leads
l1ge dual
V [())) < |[
implying that the condition (i) is satisfied.
Next, suppose (V)()), (O.) < O. Then, the condition (ii) is satisfied with M } cos (3 - -a) since
V. V.Liict + Vellctt
2
2
1.f.,, II
2
[VC(() cos (p: क //
x a) [[L(())]
2
When (V.().), (O.) < 0, the condition (ii) is also satisfied with M cos ( a).
3. DCGD (Center): the updated vector of DCGD (Center) is given by
(gf, V.((.))
gr.
dual
gi.
11,f|12
where (..))))
(P.)) (o,)ll Since ا is the angle bisector of V (0.) and (o. (see
(o.)
Proof of Proposition 4.6). th,, the angle between ((())) and gftall, is less equal to d,/2, i.c., 2/ < di < 5. From the fact that gouual is the projection of V ((e,) onto gi, we have
[[.,))
gt. اual
[VL()))) sin (v))
< [VC)))) sin
πT 0
[VL().)
sin
2
< |V
16

---

and
|ا = [V cos(1b)) (b))
VL(())] cOs
2
VV)())) cos
π -T 0
2
Consequently, the conditions (i) and (ii) are satisfied for DCGD (Center).
Remark A.I. Suppose that one employs decaying scheme for the conflict threshold a such that for example, Or. =t7, In this case, the convergence rate of
Or ((-7)) with where 0 S 7
the DCGD algorithm to a stationary point becomes O ( ( , as M in condition (ii) may depend on the conflict threshold ag For all our experiments, we set a to be fixed.
B Unification of MTL algorithms within the DCGD framework
In this section, we prove that several MTI. algorithms can understood as special cases of the DCGD framework under the PINN's formulation.
Proof. 1. MGAD 40): The updated gradient MGDA of MGDA is defined by selecting the minimum- norm element from the convex combinations of (O.) and (z,( if there is gradient confiit (O,, (O,)) < 0:
9t
MGDA
argmin ||||
s.t. u= == a2 VL. a1 + a2 = 1.
01,0220
One can easily show that (g MGDA MGDA
(gE VL,( = MGDA VLa( 2 0. Thus,
MGDA E K
Di
PCGrad [3: PCGrad uses the same update direction with DCGD (Average) when (0), (L.) O,
1 (ViLiVZz + veClictt e K
PCGrd
2
and takes (C,( (O,, when (O.), VL.(O.)) 2 0. The latter case is also contained in K;- Therefore, gt
PCGrad E K;-
3. Nash-MTL 43]: Nash-MTL considerrrsssirssssss.e... Nash bargaining solution to balance the loss gradients. NTsh-MIT. be defined by
The update gradient gt
Nush-MILL = Geve, s.t. G Gevt vi²
(B.1)
(B.2)
where G. = (O,),))). (L)) We can find the solution oy satisfying Eq. B.2 as following. By
letting Vt
11
have
02
(0.)12 (L.) VL,(0.)
(0.), V.,().))
T
U/
which is equivalent
[VV. (O), VC,(0,))o102 (3 (Co(0)))11p
(B.3)

---

Therefore, we can derive
[C-().)] [
Substituting
[V2(
[V2x V1 back into the first
equation of Eq. B.3 leads to
(0:)) + (0)ll cos(or)of
+ 01
+ cos Φe (7.)(0)))
V (0+))
+
(O.))l
01
1 cos Φe
VL,(
V..(O.) (0.)
G
I+ cOs Or
where de is the angle between (0.) and (O.)... direction with DCGD (Center). That is, Di
Thus, the update gradient si Nash-MiLL has same Nash-MIL E K;.
C Experimental details
C.1 Software and hardware environments
We conduct all experiments with PYTHON 3.10.9 and PYTORCH 1.13.1, CUDA 11.6.2, NVIDIA Driver 510.10 10 on Ubuntu 22.04.1 LTS server which equipped with AMD Ryzen Threadripper PRO 5975WX, NVIDIA A100 80GB and NVIDIA RTX A6000.
C.2 Toy example
We slightly modify the toy example in 41 to show our proposed method can expand the region of initial points that converge to the Pareto set. Consider the following loss functions with 0 (01, 02) E R
(0) (0) [2) where
(0) = 2c, (0) + c2 and L2 c; (0) (0))g(()))
f1(0) log( (0g(max(00.5(6B) 7) tanh(- 0.000005))) + 6,
f2(0) = log(max(0.5(
+ 3) tanh(-88 + 2, 2,0.000055) + 6,
(0) = ([-0, + + 0-1---02 -8)3)/10
20,
(2( = ((-0; 0.1- -8)2)/10 20.
(0)
(0) =
The landscape and contour map of above loss function are shown in Figure g The Pareto is highlighted in gray in Figure 8b We solve the above problem using ADAM, DCGD (Projection), DCGD (Average), DCGD (Center) for 100,000 epochs with different initial points. The initial points are selected as 1, 600 uniform grid points within [-10, 10] [-10, 10). Then, we mark with red dot the point at which the optimizer fails to converge to the Pareto set.
C.3 Details for Figure 2 Figure and Figure
In this experiment, we use the 7-layer fully connected neural network with 20 neurons per layer and hyperbolic tangent activation function. We train PINN models using SGD with the learning rate of 0.01 for 10, 000 epochs. addition, 100 data points are sampled boundaries and 10,000 points in the domain.
18

---

10
20
-5
10
0
-10
0
10
(a) Loss landscape
(b) Contour map
Figure The loss landscape and contour map the toy example.
C.4 Details for Section 5.1
Benchmark equations We consider Helmholtz equation, viscous Burgers' equation, and Klein- Gordon equation as the benchmark equations.
The Helmholtz equation described by
Au(x, y) + k2 u(x, f(x (x, e 2
u(x, y) = 0, (x.1) E 02,
Q = [-1,] x [-1,1].
The solution is given by u² (x, 3) = sin (A sin(T) where
f(x, v) = (k2 at - a3 π sin ( mw) sin(a)
In our experiment, we choose parameters: h: 1, 01 1, 02 4 as in 30
The Viscous Burgers' equation is given by
ur(1, uu, (1, uuza (1, 0, (x, [0. x Ω u(0, x) = sin((xx), S2
u(t, -1) u(t, 0, [0, 1).
= [1
where v 0.01
The Klein-Gordon equation is
Au(t, + yu (...) (0.7) Ω
u(0,x) = g) (=)
=0,xx) (2)
u(t,x) (t,, [0, 0n
Q = [0.
We set parameters to = 3, = 1 1 and the initial conditions, 91 (x) = x, (2(2) = for all E Ω following [30]. Then we can use the solution u² (t, x) cos(5 + (tx) where f(t, x) is derived by given equation.
We employ a 3-layer fully connected neural network with 50 neurons per layer and use the hyperbolic tangent activation function for experiments in Section 5.1 each iteration, 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points. just randomly sample the points in the boundaries if there exists an analytic solution, otherwise the points were resampled from pre-generated set for each iteration. More specifically, for the case
19

---

of Viscous Burger's equation, there is pre-determined 456 boundary points and we randomly sample in this set of points. We train PINNS for 50,000 epochs with Glorot normal initialization [48] using
DCGD algorithms, ADAM 45 LRA 30) NTK 31 35
PCGrad (37, MultiAdam
34 and DPM
We search for the initial learning rate among À = {10-3 10- 10 and use exponential decay scheduler with a decay rate of 0.9 and decay step 1, 000. For ADAM, use the default parameters: 31 = 0.9, 32 = 0.999, = 10 as in [5]. For LRA, we set a = 0.1, which is the best hyperparameter reported in 30 For MultiAdam, we use 31, 82 0.99 recommended in [34] For DPM, we test 8 = {10 10 2,10-),, ), {10-1 10 10 }, w = 1.01, 1.001}.
To compute the effectiveness of various optimization algorithms, evaluate the accuracy of the PINN solutions vu(; 0) using the relative 1e2-error defined as:
Σ 1= [u(x;; 0) u(x;) ?
Relative L error
-==1 [u(2):)22
where u(-) is the true solution and { is the set of test samples. Unless the equation has an analytic solution, method П
the numerical reference solution for u(x)) which solved by finite element
In Table report the best and worst-case relative L errors of each method across 10 independent trials (3 independent trials for two nigh-dimensional PDEs).
Equation
Helmholtz
Burgers'
Klein-Gardon
Heat (5D)
Helmholtz (SD)
Optimizer
Max
Min
Max
Min
Max
Min
Max
Min
Max
Min
ADAM
0.1053
0.0315
0.1413
0.1586
0.01985
0.0046
0.8990
0.4072
LRA
0.0108
0.0032
0.0391
0.0080
0.0166
0.0037
0.0131
0.0011
0.0991
0.0693
NTK
0,0532
0.0225
PCGrad
0,0170
0.0070
0.0322
0.0091
0.0399
0.0156
0.0149
0.0034
0.2907
0.1861
MGDA
1.0000
0,3441
1,0617
0.9037
1,0245
0,2168
1.0053
0.9577
CAGrad
0,1550
0.0330
0.0485
0.0235
0.2845
0.0872
0.0063
0.0024
1.0070
0.3070
Aligned-MFe MTI.
0.7784
0.5062
0.0640
0.0152
0.9133
0.2922
0.0018
0.0007
1.0001
0.8453
MultiAdam
DCGD (Center)
0.0038
0.0019
0.0016
0.0096
0.0112
0.0042
0.00122
0.0006
0.1007
0.0428
DCGD (Center) + LRA
0.0036
0.0013
0.0155
0.0056
0.0068
0.0036
0.0019
0.0007
0.1686
0.0515
(Center) NTK
0.0015
Table 3: Maximum and minimum of relative 2 errors in 10 independent trials (3 independent trials for two high-dimensional PDEs) for each algorithm.
We plot the exact solution, PINN solution, and its error for each benchmark equation in Figures 10 and 9
1.0
0.5
x
0.0
-025
0.030
-0.5
-030
-0.5
-050
-0.5
-075
-035
-1.0
11.0
-1
-1.0
0
0.000
X1
X1
X1
(a) Exact solution
(b) Prediction
(c) Absolute error
Figure 9: Helmholtz equation: approximated solution versus the reference solution.
20

---

1.0
1.0
1.0
5.12
0.5
08
0.5
03
0.5
0.5
530
42
5.00
0.0
X
0.0
40
0.0
8.0
0.2
0.8
1.0
0.0
0.2
0.5
0.8
1.0
-10.0
0.2
0.5
0.8
1.0
C
t
(a) Exact solution
(b) Prediction
(c) Absolute error
Figure 10: Burgers' equation: approximated solution versus the reference solution.
1.0
0.8
0.6
05
0.6
0.6
0.4
0.4
0.0
0.4
0.010
0.2
0.5
0.2
0.5
0.2
9.805
0.0
0.0
0.5
1.0
0.00
0.5
1.0
0.0
0.5
1.0
0.000
(a) Exact solution
(b) Prediction
(c) Absolute emor
Figure 11: Klein-Gordon equation: approximated solution versus the reference solution.
High-dimensional equations
We consider the following 3-dimensional Helmholtz equation
Au(z, y, 2) + k2 v, (Ix u(x, y, 2) 0, 2= 1-1
(r, E Ω (x, y. =) E an,
The solution is given by u² (zr,,))) y) = sin(a[tx) sin(a2 sin sin(a) where
f(z.y, 2) (k2 of x² a² a3 ( sin(ag sin (a5T7))
with 1,a, 4, 4, 4,03 3.
We employ a 5-layer fully connected neural network with 128 neurons per layer and use the hyperbolic tangent activation function. At each iteration, 128 points are randomly sampled in boundaries and 500 times more points in the domain as the collocation points. We train PINNs for 30,000 epochs with Glorot normal initialization.
We use initial learning rate among = 10 and use a exponential decay scheduler with a decay rate of 0.9 and a decay step 1, 000.
For 5-dimensional Heat equation, we follow the experiment setting Hao et 449 be expressed as following:
The PDE can
U kAn + f(x, t),
x EEN [0, 1]
Vu g(x, l),
E aN x [0, 1
u(x, 0) = g(x, 0),
37
where the geometric domain f = {x : |||||| S 1) and
f(x,t) t) = - || exp
(-2 |0|1 +
g(z,t) t) = exp
|=r11 +
21

---

C.5 Details for Section 5.2
Double pendulum problem Consider the double pendulum which have two point mass pendulums with masses mi, ma, two rod with length I1, l2. Let 81, Bg is the angle that the pendulums each make with the vertical and A 01 82. Set the gravitational acceleration 9.81. (see Figure 12)
Q1
1721
12
22
11/2
Figure 12: Simple double pendulum example
Then dynamics double pendulum can be described by following nonlinear differential equation system with [01, 0-1²:
[y, v" = [2(V.0')
[o, (1a) (2(1o) (to))
[wi(() (40)
subject to y(to) =
(C.1)
where
= 0
W2 82.
m2l1ww sin()20) + 2m2lgw3 sin A0 + 2gmm cos O2 sin A0 + 2gm sin 0
Si (y,,2"
21 2m + m2 sin 2 A6)
f2(y,8')) m212w3 winnnnnnn2) + 2(m + m2) sin AO + 2g(m1 + m2) cos 01 sin A6 21 m2 sin A0)
In this experiment, the initial conditions are (to)) = (to) = 6u = 150 (1(()) = we (to) = 0.
We replicate the experiment setup done in [32). We use a six-layer feed-forward network of 30 neurons on each layer with the swish activation function. We train this model with ADAM optimizer with the default hyperparameters for 20,000 epochs. We also train the same model using DCGD (Center). Figure 13 shows the training curves of ADAM and DCGD.
10
1.a
13
10-
17580
7.00
1060
7500
19600
12:00
(5:03
20160
(a) ADAM
(b) DCGD (Center)
Figure 13: Loss trajectory of each method in the double pendulum problem.
Convection equation We train PINNsFormer in Liu al. 15 to solve convection equation which can expressed as following:
": 3u, = 0, Va E [0, 2m). [0,
u(z, 0) sin(x),
u(,,t) u(2xx t)
Where 8 = 50. we follow the default setting of [15] and train the model by 500 epochs.
22

---

chotic Kuramoto-Sivashinsky equation We use causal training in Wang et al. [22 to solve chaotic equation. We use 5 layers modifed-Mi with 64 neurons per layer. and train this model 50,000 epochs for each tolerance.
+ aruul: + BUzz + yuzzan 0, Vor E [0, 2T), E [0,0.5] u(0, x) = cos(z)( sin(x))
where a 100/16, 100/162,1 1 100/164.
0.0
1.00
0.0
100
0.0
5.00
0.75
0.75
0.2
0.2
0.2
0.50
030
0.4
0.25
0.4
025
9.00
0.00
-025
-0.50
..50
-0.55
-075
1.0
-1000
X
(a) Exact solution
(b) Prediction
(c) Absolute error
Figure 14: Convection equation: approximated solution versus the reference solution.
1.0
2.00
1.0
1.75
1.75
0.02
1.50
1.50
0.01
1.25
3.25
9.01
1.00
1.00
5.00
0.75
0.75
5.00
0.2
050
0.2
0.50
5.01
025
0.75
5.00
0,0
0.00
0.0
0.00
0.0
0.00
X1
X1
X1
Exact solution
(b) Prediction
(c) Absolute error
Figure 15: 2D- 2D-Volterra equation: approximated solution versus the reference solution.
Auxiliary-PINe PINN: Nonlinear integro-differente equation A-PINN is a variant of PINNs, de- signed to solve cinngro-differentaee equations [9]. We apply our DCGD algorithms A-PINN for solving the following nonlinear 2-dimensional Volterra IDE:
SZ x) Dt2
Ōu(t, x) dx
@ult, x) ult,, x) + g(t, x) + l- S/' cos(v1 - y2) 82 32) dy2
Øt
where the boundary conditions are u(O, x) = x, au(0.z) sin and u(t, 0) = t sin(t), 0 S t, x S
1. The analytic solution is u" (t, + sin + x) with À where g(t, 2)) is derived by given equation.
Within the framework of A-PINN, it converts the above integral equation into the following equation by representing integrals as auxiliary output variables:
Pu(t, x) du(t, du(t,z) u(t, + g(t, x)) + xo(t,
012
Or Ø1
@c(t,x) dr 1' cos(y) x)u(v,, x)dy tw(t,x))
@w(t, x) cos(t - x))(t,
àt
where the new variables v and satisfies the boundary condition v(t, 0) 0. w(0, 0.
23

---

For A-PINNs, we employ 3-layer fully connected neural network with 50 neurons per layer and hyperbolic tangent activation function. For training. 128 points are randomly sampled in boundaries and 10 times more points in the domain as the collocation points in each epochs. We train A-PINN models for 5, 000 epochs.
Separable PINN:
3-dimensional Helmholtz equation SPINN a novel architecture designed to effectively reduce the computational cost of PINNs, especially when addressing high-dimensional PDEs [7]. To test the performance of DCGD for SPINNs, we consider the following 3-dimensional Helmholtz equation
Au y, k2 u(,, 2) (,),,; (0,11 E Ω
y: 2) =0 0, (I, E 0N,
a 1-11-11]3
The solution is given by u² (xx y) = sin(ai sin(a sin(ag where
f(z. v. 2) = (k²2 - a² a² aं sin sin(ay sin
with A: = 1, = 4, = 4, ag = 3.
We follow the optimal hyperparameter setting reported in [17]. For ADAM and DCGD (Center), the learning rate 0.001. The input points are resampled every 100 epochs. Regarding model architecture, use the SOTA model, called (SPINN + Modified MLP). We record the mean and standard deviation of relative L errors from 3 independent trials in Table indicating that the performance of SPINN can be significantly improved when trained with DCGD for varying number of collocation points.
Method
N₂
Relative L error
Training speed
163
0.0578 (0.0039)
1.65 (ms/iter)
323
0.0352 (0.0035)
1.78 (ms/iter)
SPINN
643
0.0280 (0.0066)
2.38 (ms/iter)
1283
0.0294 (0.0123)
2.71 (ms/iter)
256
0,0319 (0.0026)
5.12 (ms/iter)
163
0.0447 (0.0176)
1.76 (ms/iter)
323
0,0104 (0.0048)
1.90 (ms/iter)
SPINN DCGD (Center)
64
0,0032 (0.0002)
2.59 (ms/iter)
1283
0.0015 (0.0003)
2.85 (ms/iter)
2563
0.0020 (0.0009)
5.34 (ms/iter)
Table 4: Helmholtz Equation (3d): Relative L errors and training speed. No is the number of
collocation points.
(a) Exact solution
(b) Prediction
(c) Absolute error
Figure 16: 4D-Helmholtz equation: approximated solution versus the reference solution.
24

---

D Supplemental results
D.I Ablation study
In Section we introduce three specific algorithms: DCGD (Projection), DCGD (Average), and DCGD (Center). We conduct an ablation study to investigate the impact of different updated gradient schemes within the dual cone region. More specifically, we compare the performance of these three algorithms on five benchmark equations. Detailed experimental settings can be found in Appendices C.4
Tables 5 and 6 demonstrate that DCGD (Center) outperforms the other DCGD algorithms across all experiments. Therefore, we consider DCGD (Center) as the default DCGD algorithm.
Equuation
Helmholtz
Burgers'
MeinnGooddon
Heat (SD)
Helmholtz (3D)
Optimizer
Mcan (std)
Mean (std)
Mean (std)
Mean (std)
Mean (std)
Projection
0.0089 (0.0022)
(0.0035)
0.0216 (0.0130)
0.0074 (0.0049)
0.1251 (0.0044)
Average 0.0166 (0.0124)
0,0156 (0.00322
0,0292 (0.0088)
0.0051 (0.0008)
0,3161 (0.0911)
Center
(0.0005)
(0.0046)
(0.0027)
(0.0003)
(0.0250)
Table 5: Average and standard deviation of relative 1,2 errors in 10 independent trials (3 independent trials for two high-dimensional PDEs) for each DCGD algorithm.
Equation
Helmholtz
Burgers'
Klein-Gordon
Heat (SD)
Helmholtz (3D)
Optimizer
Min
Min
Max
Max
Projection
0.0138
0.0062
0.0217
0.0097
0.0573
0.0120
0.0144
0.0036
0.1286
0.1
Average
0.0469
0.0078
0.0209
0.0115
0.0442
0.0178
0.0060
0.0040
0.3942
0.1883
Center
0.0038
0.0019
0.0163
0.0096
0.0112
0.0042
0.0163
0.0096
0.1007
0.0428
Table 6: Min and Max Relative 2 errors in 10 independent trials (3 independent trials for two high-dimensional PDEs) for each DCGD algorithm.
D.2 Computational cost
In this section, we acknowledge that our proposed method incurs higher computational costs due to the need for backpropagation for each individual loss. Nonetheless, through a comparison of training speeds, we empirically demonstrate that DCGD achieves superior performance with computational costs comparable to those of existing competitors.
PDE equation
Optimizer
Heat (5D)
Helmholtz (3D)
ADAM
11.1 (iter/s)
9.05 (iter/s)
L-BFGS
0.53 (iter/s)
1.19 (iter/s)
LRA
3.39 (iter/s)
5.43 (iter/s)
NTK
3.92 (iter/s)
7.49 (iter/s)
MultiAdam
4.10 (iter/s)
6.85 (iter/s)
PCGrad
5.98 (iter/s)
8.94 (iter/s)
MGDA
3.46 (iter/s)
6.16 (iter/s)
CAGrad
4.22 (iter/s)
8.80 (iter/s)
Aligned-MTT.
4.10 (iter/s)
7.97 (iter/s)
DCGD (Average)
3.78 (iter/s)
5.42 (iter/s)
DCGD (Projection)
3.70 (iter/s)
5.64 (iter/s)
DCGD (Center)
4.35 (iter/s)
8.90 (iter/s)
Table 7: Training speed in higher dimensional equations example
25

---

E Pseudo codes of algorithms
This section provides pseudo codes for the proposed DCGD algorithms.
Firstly, DCGD (Projection) uses the projection the total gradient VL(0)) onto G, when VL( &
Kj. Otherwise, (L(()) is used. Then the update vector gh can be defined follow:
V VC((e),
if ((,)) E K if if ((,)) K; and (VL(().) (O. < 0)
DCGD (Projection)
V.LIIEE
if LL()) K and (VL())), (L,(0.) < 0)
(E.1)
Secondly, DCGD (Average) uses the the average of ViLi and V.Live when the total gradient is outside K; Otherwise, V.(())) is used. The update vector اسسوو of DCGD (Average) is defined as follow:
dual VC((.) (VV.LTVE + VoLlvet)) if VL()) & K2
if VC E K;
DCGD (Average)
(E.2)
Thirdly, DCGD (Center) employs the following update vector gtt!. regardless of whether the total gradient V ((.))) included K;:
fef, VL((O)) || Di where of (Lr(O.)
(O) VO,1
DCGD (Center)
St
(E.3)
VL.(
Here, pseudo codes of these algorithms are summarized in Algorithms introduce conflict threshold as stopping condition for DCGD algorithms, as they can reach Pareto-stationary point characterized by Φe That is, the algorithm stops when the parameter converges close to Pareto-stationary point such that cos(())) - < a. Throughout our experiments, set 10-8
and Note that we
Algorithm 2 DCGD (Projection)
Require: learning rate A, max epoch T, initial point Co. gradient threshold e, conflict threshold for t to T do if break
Φ2 < x or [VL(())]| <
then
end if
if (0,)) & K then
gt duol (((,))
else if v L(0,) K* and (VL(),), VL,(O.) < 0 then
guual = VeLivet
else if (L(,) K" and (VL())), (L.( < 0 then
dual V
end if end for
DCGD algorithms be easily combined with other optimizers strategies thanks its flexible framework. For example, one can design a DCGD algorithm combined with ADAM to leverage advantages of adaptive gradient methods, see Algo. 5 For our experiments, consider the DCGD (center) combined with ADAM as the default. Algorithm 6 presents the psedo code for DCGD combined with loss balancing method such LRA and NTK.
26

---

Algorithm 3 DCGD (Average)
Require: learning rate X, max epoch T, initial point 6. gradient threshold E, conflict threshold for f = to T do
if
P2 S x or [V < then
break
end if
if (L(:)) & K" then
geual V + V
else
9t duot v((O))
end if
0: 0c-1- daptual end for
Algorithm 4 DCGD (Center)
Require: learning A, max epoch T, initial point Oo. gradient threshold e, conftict threshold for = 1 to T do
if
0z S x or [VV(()))] then
break
end if
1 + (C.(
(c(().)
gi
(A))l
91.
dual
(gf. V((b.)
1 oi
0, 0:-1 Agdual
end for
Algorithm 5 DCGD with ADAM
Require: learning rate X, max epoch T, betas $1, DCGD operator DCGD(.)
for to T do
91 dual (CG) (O))
9: - gdual
m, t- 81m--1 + (1 31 19t
< B2t1-1 + -B2)
my t- my
1-
0, -- 0,-1 7it viete
end for
Algorithm 6 DCGD with a loss balancing method
Require: learning rate X, max epoch T, loss balancing operator LB(-)
for = to T do
(8,, (6): LB (0.), (o.))
(,( < (0,)
(0,) + B, L. (0:)
Choose gduil E K
0, 4.=00 -0,-1 Ag dual
end for
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes] Our abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes] We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes] We have stated the explicit assumptions and provide complete proofs for main results in Appendix[A]
Guidelines:
28

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes] The pscudo code of our algorithms can be found in Appendix E and we have stated detailed instructions in Appendix C We have also provided code to reproduce the results of main experiments in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes] We have provided our code to reproduce the results of main experiments in supplementary material.
29

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc.
public/guides/ Stttle///coSSubbbssiiooooP0
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wwwoddesubissssooooolllooooo
//nips.cc//ppp mtaips.cc/publiccgui.eee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes] All experimental setting is explained Appendix Cinlluding hyperparame- ter choices, training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We repeated the main experiments in several times and reported several statisties including min, max, and the standard deviations for our numerical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
30

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] See Application C.1 for details of resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ het::://neuriiee httpr//pp.../pplllc/ccccccceeeee
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
31

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes] The overall experiments followed the existing experimental setup which have cited, and the code and data are available for use under the MIT license. For CausalPINNs in [22]. the code is available under the CC-BY-NC-SA 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpssspppppaittteeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
32

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "RE7msrD7wY", "reviewer": "Reviewer_Nfs2", "summary": "PINNs have become commonplace in both scientific computing and machine learning communities and have found widespread applications. There have been many modifications to various aspects of PINNs, such as the loss functions, initialisations, initial and boundary condition representations, learning algorithms and network architectures. \nA major problem with learning effective solutions through PINNs relative to other learning tasks is their difficulty in converging to the actual solution due to the complex nature of the underlying physics and equations involved and the corresponding loss landscape. \nThe authors have provided a comprehensive survey of the previous works, making the unaddressed problems and goals very clear. A novel algorithmic framework based on convex optimization and geometric analysis is presented, which demonstrates superior performance on many tasks. \nThe contributions of this paper are multifold, one being a new optimization algorithm while the other being a unified, open and flexible framework to adapt, modify and integrate the current technique with previous and future techniques.\nThe paper is well written with enough information on the theoretical and mathematical underpinnings, experimental setup, implementation details, results and justifications.", "questions": "I have gone through the previous works mentioned in detail, and does seems like the problem of multi-objective optimization or even optimization, specifically in the case of PINNs, has not been looked at from the perspective of Dual Cones. Could the authors add some more works related to Dual Cone-based techniques to the previous/related works section?\n\nTypos and presentation issues are already covered in the weaknesses section; however, here, I would like to mention a few pointers where I would like to seek clarification. (This is more from a clarification perspective for me individually than a comment on the work, though I believe these clarifications will help the broader community)\n\nAppendix A, line 779, Proof of Theorem 4.5: In the derivation of line779, from the first step to the next, should there be an equality sign in place of the first inequality sign as we are just distributing the dot product over addition on the first term $\\nabla \\mathcal{L}(y+t(x-y))$? (i.e. the first of the 3 inequality signs should be an equality instead)\n\nAppendix A line 797: Is the expression $\\left| \\cos(\\phi_t) - \\pi \\right| < \\alpha$ is correct and as intended? The inequalities on lines 810 and 811 and the threshold criterion in the algorithms in Appendix E ($\\pi - \\alpha < \\phi_t \\leq \\pi$) point otherwise. I believe there should not be the cosine of $\\phi$; rather, it should just be the angle/phase difference $\\phi$.\nI have a similar comment for line 984, as the same expression is used.\n\nAppendix C.5 line 925: Is the statement complete? Is it intentionally left this way?", "limitations": "I am appreciative of the authors who have mentioned (and also acknowledged in the checklist) a few limitations of the work in section 6 this is more from a future work perspective than from a limitations of the work presented as part of this paper. The limitations of the current work are not very evident from the present text.\n\nIt would be further helpful if they could include a more detailed section on where this cannot be applied or other multi-objective scenarios where the DCGD optimization framework is not as effective as other previous techniques.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Although the concept of dual cone is well known in the optimization domain, specifically in the convex optimization area, the authors utilise and combine existing techniques from optimization theory in a novel way to develop a new algorithmic framework, which they refer to as DCGD. \n\nThe authors propose the DCGD algorithm which involves picking a gradient from the Dual Cone region formed by the two losses (boundary loss and residual loss) of PINNs and also theoretically prove that it converges to a pareto-stationary point. Thus, along a new algorithm, theoretical guarantees for convergence are provided, which are very crucial for the goal of improving the training convergence of PINNs.\n\nWith a suitable candidate being chosen as the new gradient from an explicitly defined space (called here as $G_t$), multiple variants of the DCGD algorithm can be created. The DCGD can also be combined with many existing optimization techniques, such as Adam Optimizer and Neural Tangent Kernel (NTK). \n\nThis work introduces a fairly novel optimization technique which has huge potential for applications in PINNs and multi-objective optimization. The authors have provided strong justification for the generality of their work by showing that most previous MTL approaches are special cases of DCGD. The general, modular and extensible nature of the work paves the way to significantly accelerate further research in various allied areas.\n\nThe supplementary material includes the datasets and code for all the experimental details mentioned in the paper and the appendices. I have glanced through the code and data files provided but have not personally verified the results by running the code. (might update here if I get to install and run the code provided)", "weaknesses": "The authors have provided enough empirical and theoretical evidence for their proposed DCGD method, though since this work proposes a theoretical and algorithmic framework, albeit it would be notable to have a section on comparing the computational complexity and qualitative terms with existing algorithms. I would say a short section in Appendix D/E would go well.\n\nSince the problem at hand is of multi-objective optimization which overlaps with multi-task learning problems and PINNs are a natural suitable choice, it would be very helpful to see the results of this approach on PINOs (Physics Informed Neural Operators). There is no comparison (even qualitative) of DCGD on PINOs (There are PINOs which are trained only on residual loss, but here I'm specifically referring to PINOs which include both residual and boundary loss for being relevant to DCGD). Was this not considered or omitted, or is it expected to show improvement in performance, as PINNs have shown? (Personally, I believe it should show improvements as the authors have already demonstrated superior performance on PINNformer with DCGD)\n\n\nPresentations/Grammar/Typos (I do understand that the authors would be well aware of these and would rectify these in the pre-print or camera-ready versions. However, I would like to mention them here together to aid them. Some might have been missed; might update if needed)\n\nEquation 2.2: The weight of the boundary loss is mentioned as $\\omega_u$ and also on line 80, whereas it is mentioned as $\\omega_b$ (which seems more appropriate) on line 79\n\nSection 4.3: (This is a nitpick and can be conveniently ignored)\n$g_t^c$ could be written on the same line as eq. 4.3 or as a newline as eq. 4.4 similar to how equation E.3 is written\n\nSection 5.2: Is the title intended to be “P(I)DEs” or “PDEs”? I believe this is just a typo\n\nAppendix A, line 769: Equations A.8 and A.9 could be mentioned here (as A.4 and A.5) and then referenced in the Proof of Proposition 4.6 section. I believe this will greatly enhance readability\n\nAppendix A: Proof of Theorem 4.2 line 757: The residual loss is repeated twice $\\nabla\\mathcal{L}_r(\\theta_t)$ is repeated twice, it should be both boundary and residual loss, $\\nabla\\mathcal{L}_r(\\theta_t)$ and $\\nabla\\mathcal{L}_b(\\theta_t)$\n\nAppendix A, line770: \"One can derive that …\" should we include the boundary loss term instead of residual loss as the derivation with residual loss is already shown in line769 i.e it should be $\\langle g, \\nabla L_b (\\theta_t) \\rangle \\geq 0$\n\nAppendix A, line 781: For the sake of notational consistency, could you replace \"condition (2)\" with \"condition (ii)\"\n\nAppendix A, line 785: (nitpick) The subscript t is omitted in the boundary loss and included in the residual loss ($\\nabla\\mathcal{L}_r(\\theta_t)$ and $\\nabla\\mathcal{L}_b(\\theta)$). I suggest the subscript \"t\" be included at both the places for notational consistency\n\nAppendix A, line 818, Proof of Corollary 4.7:  3. DCGD (Center): “vector” instead of “veoctr”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your  insightful comments, and for thoroughly checking details such as typos. We deeply appreciate your high-quality review.\n\n- **Response to Weaknesses (W)**\n\n> **W1. Comparison of Computational Complexity**\n\nPlease refer to the computational complexity discussion in our global response.\n\n> **W2. Application to PINOs**\n\nThank you for your valuable suggestion regarding the application of DCGD to PINOs. We have conducted additional experiments on PINOs for Burgers' equation and Darcy flow. The results are summarized in **Table 1** of the attached PDF, which confirm that DCGD can also enhance the performance of PINOs. For further details on the broader implications and potential impact beyond PINNs, please refer to our global response.\n\n> **W3. Typos**\n\nThank you for carefully checking the details. We will incorporate the suggested corrections into the revised manuscript. One point to clarify: “P(I)DEs” refers to partial integro-differential equations.\n\n- **Response to Questions (Q)**\n\n> **Q1. Previous Work Related to Dual Cone-Based Optimization**\n\nDual cones are a fundamental concept in optimization, particularly in convex optimization. They are used to characterize optimal solutions and feasibility conditions in constrained convex optimization and duality theory. However, to the best of our knowledge, there have been no previous instances of incorporating dual cones into gradient manipulation or developing a dual cone-based gradient descent method. We emphasize that our proposed framework is not merely an application of existing methods to multi-objective problems or PINNs but represents a completely novel development in the field. For further discussion on the novelty and originality of our work, please refer to our global response.\n\n> **Q2. Derivation of line 779**\n\nYou are correct. We perform the distributive property of the inner product, which results  in an equality. Thus, the inequality still (obviously) holds, but for clarity, we will change it to an equality as suggested.\n\n> **Q3. Line 797**\n\nThank you for your comment. The threshold $\\alpha$ is set to stop the algorithm when it reaches near the Pareto-stationary point. Therefore, the cosine should be removed from the equation, i.e., the expression $\\vert \\cos(\\phi_t) - \\pi \\vert < \\alpha$ should be changed to $\\pi - \\alpha < \\phi_t \\leq \\pi$ . \n\n> **Q4. Line 925.**\n\nThank you again for your careful review. The sentence that should have been removed during the annotation process was accidentally included.\n\n- **Response to limitations**\n\nWhile this paper demonstrates the valuable result that DCGD converges to Pareto-stationary points, which is a notable advantage over popular optimizers like SGD and Adam, there are still unexplored aspects regarding its convergence. For instance, an important and challenging question is whether DCGD can be guided to converge to a better Pareto-optimal point within the Pareto set. Additionally, although we proposed three algorithms—Center, Projection, and Average—within the DCGD framework and found DCGD (Center) to be the most effective for PINNs, there may be more efficient algorithms for other problems (e.g., PINO, MTL). These possibilities were not addressed in this paper and represent interesting directions for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAXGmJlEUM", "reviewer": "Reviewer_Bm5Z", "summary": "In this paper, it is identified that PINNs can be adversely trained when gradients of each loss function exhibit a significant imbalance in their magnitudes and present a negative inner product value. To address these issues, the authors propose a novel optimization framework, Dual Cone Gradient Descent (DCGD), which adjusts the direction of the updated gradient to ensure it falls within a dual cone region. This region is defined as a set of vectors where the inner products with both the gradients of the PDE residual loss and the boundary loss are non-negative. Theoretically, the convergence properties of DCGD algorithms in a non-convex setting are analyzed. On a variety of benchmark equations, DCGD outperforms other optimization algorithms in terms of various evaluation metrics, achieves superior predictive accuracy and enhances the stability of training for failure modes of PINNs and complex PDEs.", "questions": "Line 277, \"Failure model\" vs. \"Failure modes\"?", "limitations": "The proposed Dual Cone Gradient Descent (DCGD) optimization method for PINNs may still be trapped in local minima.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical analysis is provided, including the necessary and sufficient conditions under which the total gradient falls within the dual cone region, and the convergence rate.\n2. Superior performance.", "weaknesses": "1. It is better to provide the visualization of PINN predictions and compare with exact solutions, especially for the failure modes of PINNs.\n2. Training timing is not reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and suggestions.\n\n- **Response to Weaknesses (W)**\n\n> **W1.Visualization of PINN predictions and exact solutions.**\n\nPINNs predictions and the exact solution, along with absolute error plots, can be found in Appendix C of the original manuscript. Please refer to Figures 9-15 of the original manuscript.\n\n> **W2. Training time**\n\nWe provided a comparison of the training speed of the proposed method with other optimization algorithms in Appendix D.2 of the original manuscript. For additional details on the computational cost, please refer to our global response.\n\n- **Response to Questions (Q)**\n\n> **Q1. Line 277**\n\nThank you for the correction.\n\n- **Response to Limitations**\n\nIt is indeed a common challenge in high-dimensional nonconvex optimization problems for all optimization algorithms to potentially get trapped in local minima. However, DCGD has a significant advantage in that it not only converges to stationary points but also to Pareto-stationary points, distinguishing it from methods like SGD and Adam that only guarantee convergence to stationary points. This distinction is a key contribution of our work, as demonstrated in Theorem 4.5. Additionally, the empirical results from the toy example shown in Figure 6 effectively support this theoretical result.\n\nMoreover, integrating noise injection techniques, such as Stochastic Gradient Langevin Dynamics (SGLD) [1], could further enhance the exploration of better local minima, although we have not investigated such variations in this paper. \n\n[1] W. Max and Y. Teh. \"Bayesian learning via stochastic gradient Langevin dynamics.\" ICML, 2011.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uqtg3C88WG", "reviewer": "Reviewer_NFZX", "summary": "The paper concerns with the training of physics-informed neural networks (PINNs). It observes that in a multi-objective setting a naive gradient update which decreases the total objective might not decrease every individual objective. This is used to explain the challenge of training PINNs which have a potentially conflicting PDE and a boundary-condition loss. As a remedy, the work proposes to find another descent direction which decreases both of the PINN losses. Three different methods are proposed and the superior \"center\" method is selected for more detailed empirical investigation. It demonstrates strong results relative to several existing PINN optimizers.", "questions": "- I would kindly ask the authors the clarify Theorem 4.5 in terms of the aforementioned concerns.\n- How do the aforementioned optimization strategies compare to your experiments? Do you foresee a way for your method to be compatible to these (similar to section 5.3)?\n- Does your method extend to more than two losses? Even in PINNs, there might be additional losses for data and initial conditions.\n- How important is the assumption of fixing $\\omega_r = \\omega_b = 1$? Do the other optimizers show improved performance for other choices of these loss weights?", "limitations": "The limitations are discussed rather sparingly and should also include that the method currently applies to just two losses. It should also be noted that the performed experiments are closer to toy than practical problems. \n\n**Errors and suggestions**\n- Line 59: It would be helpful if you briefly described what each of these three approaches do, instead of only listing examples.\n- Line 73: $d$ is used for both the dimension of the domain as well as the parameter space.\n- Eq. 2.2 and line 88: $\\omega_u$ should be $\\omega_b$\n- Line 79: should this be an equality?\n- I do not think writing out algorithm 1 adds much value as this is precisely gradient descent but with a different gradient.\n- Line 170: should there be an argmin or arginf instead of inf? Or alternatively $\\mathcal{L}(\\theta*)$ instead of $\\theta*$\n- Lines 229-240 and Figure 6 might better belong to the appendix as it detracts attention from the main story.\n- Line 185: \"presents\" -> \"present\"\n- Line 221: \"ADAM\" -> \"Adam\"\n- A.4: the first inequality should be an equality\n- Adjust some odd wording, e.g. \"mysterious challenges\" (line 44), \"harmoniously\" (line 48)\n- I suggest to moderate the claims about the practical success of PINNs (see weaknesses for concrete examples).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a clear central story that explains multi-objective optimization in terms of simple geometric concepts. The theoretical analysis in addition to the empirical evaluation is a welcome addition.", "weaknesses": "However, the significance of the theoretical contributions must be questioned. There are just two theorems, of which 4.2 is fairly trivial and 4.5 is unclear: why is $\\frac{1}{T+1}$ on both sides of Equation 4.2, and what does this equation even state? I cannot make sense of the provided intuition that the convergence rate is $\\mathcal{O}(1/T)$, i.e., the more iterations, the slower you converge. Even more so, it is unclear how this intuition follows from Eq. 4.2 in the first place: it says that the sum of gradient norms is upper-bounded by the gap from the initial to the optimal loss, up to some constant.\n\nThe work also overlooks some recent literature explaining the difficult behavior of PINNs in terms of poor conditioning and proposing strong optimization strategies, which to the best of my knowledge achieve state-of-the-art convergence. I would argue these ideas should be included in the related work and as baselines contributing to a much more complete story.\n- *De Ryck et al. An operator preconditioning perspective on training in physics-informed machine learning. ICLR 2024.*\n- *Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.*\n\nSeveral presentation choices could be improved, including minor mathematical and typing errors, odd wording, and superfluous parts (see suggestions below).\n\nThe work also overstates the practical success of PINNs: \"a prominent approach for solving PDEs\" (line 2), \"remarkable empirical performance\" (line 4), \"PINNs have achieved great success in a wide range of applications\" (line 8). This sets a misleading context for the practical significance of PINNs and by extension the presented work.\n- *Grossmann et al. Can Physics-Informed Neural Networks beat the Finite Element Method? IMA Journal of Applied Mathematics 2024.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and for raising questions from the perspective of readers who may be less familiar with optimization theory. We hope the following responses will clarify any misunderstandings and address your concerns.\n> **Response to W1 (Theorem 4.2. is trivial).**\n\nWe respectfully disagree with the assessment that Theorem 4.2 is trivial. We believe that this theorem is not only intuitive but also provides significant insights. As detailed in Section 3, much of the existing literature on PINNs has explained gradient pathologies using concepts like gradient conflict and dominating gradients. However, dominating gradients have been somewhat ambiguously defined, often leading to ad hoc strategies. Theorem 4.2 clarifies that dominating gradients should be adaptively defined based on the angle between gradients, addressing a fundamental gap in understanding gradient pathologies that has been overlooked in previous work.\n>**Response to W2 and Q1 (Interpretation of Theorem 4.5).** \n\nFirstly, the statement of Theorem 4.5 (equation 4.5) is standard in optimization theory for showing the convergence of optimization algorithms (for example, see Corollary 2.2.1 of [1] and Theorem 3.2 of [2]).\n\nAdditionally, your interpretation of \"the more iterations, the slower you converge\" is incorrect. The left-hand side of equation 4.5 represents the average of the squared gradient during training, and it implies that:\n$$\n\\min_{t=0,1,\\ldots, T}\\|\\nabla\\mathcal L(\\theta_t)\\|^2\\leq\\frac{1}{T+1}\\sum_{t=0}^T\\|\\nabla\\mathcal L(\\theta_t)\\|^2=\\mathcal O(1/T)\n$$\nThus, the optimization algorithm converges to a stationary point such that $\\|\\nabla\\mathcal L (\\theta_t)\\|=0$ as the number of iterations increases. This result represents the optimal convergence rate that can be achieved under the condition that only the gradient being Lipschitz continuous is provided.\n\nFurthermore, Theorem 4.5 provides crucial information compared to the convergence results of existing gradient descent methods like SGD and Adam. While SGD and Adam are only guaranteed to convergence to a stationary point, DCGD not only converges to stationary points but also has the potential to converge to Pareto-optimal solutions in multi-objective optimization.\n\n[1] Zhuang, J, et al. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. NeurIPS 2020.\n\n[2] Hazan, Elad. Lecture notes: Optimization for machine learning. 2019.\n> **Response to W3 and Q1 (Comparison with Two Recent Papers [3,4])**\n\nThank you for introducing the recent papers. Since one of these papers was published after our submission, we were unable to include it in our related work at that time.  After carefully studying the papers, we compare the motivation, theoretical aspects, and performance of the papers with ours:\n\n**Motivation:**\n\nBoth papers focus on the ill-posedness of PINN loss and discuss the importance of preconditioning and the combination of second-order and first-order methods. In contrast, our work analyzes PINN gradient pathology from a multi-objective optimization perspective and proposes a novel algorithm to address these issues.\n\n**Theoretical Aspects:**\n\nWe kindly point out that the review's statement that “the convergence results in the two papers are state-of-the-art” is somewhat inappropriate because their convergence results cannot be directly compared to ours. The papers derive convergence results for optimizers by making assumptions about the Hessian (PL condition) and the overparameterized model (assumption that the model can achieve zero loss on the training dataset, see assumption 8.1 of [4]).\n\nIn contrast, our results are derived under **the minimal assumption** that gradients are Lipschitz continuous, without any assumptions on the Hessian or the overparameterized model. Thus, the theoretical settings and assumptions are completely different, making a direct comparison of theoretical results inappropriate. To reiterate, our derived convergence rate represents the optimal rate that can be achieved when the Lipschitz continuity condition of the gradient is only provided.\n\n**Performance:**\n\nOur paper and the two mentioned papers take completely different angles on PINN optimization. Thus, combining these ideas harmoniously could lead to further improvements. Indeed, our additional experiments show that the combination of NNCG from [4] with DCGD results in improved performance, see **Table 4** of the attached PDF.\n\n[3] De Ryck et al. An operator preconditioning perspective on training in physics-informed machine learning. ICLR 2024.\n\n[4] Rathore et al. Challenges in Training PINNs: A Loss Landscape Perspective. ICML 2024.\n\n>**Response to W4. overstatement of the practical success of PINNs**\n\nThank you for pointing this out. We are happy to adjust our language to be more neutral in the revised manuscript.\n>**Response to Q4 ($w_r=w_b=1$)**\n\nThe setting $w_r = w_b = 1$ is a common choice in the PINN literature for simplicity in presentation (e.g., [5, 6, 7] ). Furthermore, since DCGD updates using a dual cone region to avoid gradient conflicts across different values of $w_r$ and $w_b$, this ensures robust performance of DCGD regardless of the specific values chosen for $w_r$ and $w_b$. To support this, we have compared the performance of optimizers under different values of weights. As shown in **Table 3**, DCGD consistently demonstrates robust and superior performance compared to other competitors across different values of $(w_r, w_b)$.\n\n[5] Raissi, M., et al. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics.\n\n[6] Wang, S., et al. (2021). Understanding and mitigating gradient flow pathologies in physics-informed neural networks. Journal on Scientific Computing.\n\n[7] Zhao, Z., et al. PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks. ICLR 24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cuWpDEvF69", "reviewer": "Reviewer_SdDT", "summary": "**Summary:** The paper proposes viewing the optimization of PINNs as a multi-objective optimization problem, with boundary and residual terms as potentially conflicting objective functions. To address conflicting gradients in the optimization process, the dual cone of the cone generated by the gradients of the boundary and residual terms is designated as the set of non-conflicting update directions. Several algorithms that guarantee updates from this dual cone are proposed, and their empirical performance is assessed.\n\n**General Impression:** The manuscript is well-written, easy to follow, and generally of high quality. The numerical experiments are convincing and demonstrate the good performance of the proposed method. However, I disagree to a certain extent with the multi-objective viewpoint for the optimization of PINNs. For well-posed forward problems, the boundary loss and the interior loss are not really conflicting losses—at least if we assume that the network ansatz is sufficiently expressive. I expand more on this issue below in the questions. Nevertheless, the preprint presents a convincing suite of numerical experiments that seem to benefit from the multi-objective optimization approach.", "questions": "**Questions**\n\n1. It is unclear whether PINN optimization should be viewed as a multi-objective optimization problem. Please comment.\n2. I am missing a clarification on whether the proposed dual cone gradient descent is a completely novel idea, even in the field of multi-objective optimization, or if it is well-known in that field and its application to PINNs is novel. Please clarify and give references if appropriate.\n3. Could the authors comment on how their proposed method relates to loss reweighting strategies? It seems that the proposed approach could equivalently be understood as a loss reweighting strategy.\n4. Can the authors comment on the case of more than two loss functions.\n5. Have the authors considered measuring angles in function space instead of parameter space by using the differential of the parametrization map? Recently, optimization methods that employ function space geometry have shown promise for the optimization of PINNs (see [1, 2, 3]). This might lead to further improvement of the proposed method.\n\n**Details to question 1:**\nA fundamental question for me is whether to view PINNs as a multi-objective optimization problem or not. To illustrate this question, consider the following academic example of a PDE with homogeneous forcing and inhomogeneous boundary data:\n\n\\begin{align}\nNu &= 0 \\quad \\text{in }\\Omega,\n\\\\\n\\newline\nBu &= g \\quad \\text{on }\\partial\\Omega.\n\\end{align}\n\nAssume further that $N$ maps constant functions to zero (any PDE operator without a zeroth-order term). Assume a trial neural network is initialized representing (almost) the zero function (or any other constant function). Any change in the neural network will now likely increase the PDE residual (which is at its global minimum for any constant function). Furthermore, the increase in the PDE residual is necessary to obtain the correct solution. This might lead to a Pareto stationary point that has nothing to do with the sought-after solution.\n\n**References**\n\n[1] https://proceedings.mlr.press/v202/muller23b/muller23b.pdf\n\n[2] https://openreview.net/pdf?id=z9SIj-IM7tn\n\n[3] https://arxiv.org/pdf/2402.07318", "limitations": "See questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths:**\n\n1. Considering the dual cone of the interior and residual gradients is a simple and sound idea. The presentation is clear and understandable.\n2. The section on numerical results demonstrates the benefits of the proposed method and incorporates many recently proposed techniques for the optimization of PINNs.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and positive feedback.\n\n> **Response to Q1.**\n\n**Simple Answer**: As discussed in Section 3, it is empirically observed that gradient conflicts and imbalances between losses frequently arise during PINN training, which has led to the development of various loss balancing strategies. Our work is motivated by the idea that viewing and addressing these issues from a multi-objective optimization perspective can offer a more effective solution.\n\n**Long Answer**: We appreciate the depth and challenge of this question, as well as the detailed example you provided. We think this question is closely associated with a fundamental aspect of the PINN learning process, which is indeed complex and not yet fully understood both theoretically and empirically.\n\n\nThe trade-off between increasing PDE residuals and achieving better approximations to the true solution is indeed a critical question, and there is no theoretical guarantee that this trade-off will always lead to improvements. In the example you mentioned, even if the network is initialized at the trivial solution 0, we think it is possible for the network to reach the true solution smoothly without increasing PDE residuals. This is because, in the absence of boundary conditions, there are infinitely many solutions that satisfy $\\mathcal N u=0$, including solutions other than the trivial one. The neural network can transition smoothly among these solutions and approach the true solution.\n\n\nIn this context, the main goal of DCGD is to facilitate a smooth learning process without sacrificing PDE residuals. This approach is particularly important in failure modes like the double pendulum example in Section 5.2, where maintaining a smooth and effective learning trajectory is crucial.\n\n>**Response to Q2.**\n\nThank you for the opportunity to clarify the novelty and originality of our DCGD framework. While the concept of dual cones has been used in convex optimization for duality theory and formulating optimality conditions, our paper employs dual cones to characterize a space where gradient conflicts do not occur and integrates this with gradient descent methods. This approach is novel not only in the context of PINNs but also within gradient descent methods more broadly.\n\n\nFor a more detailed discussion of the novelty and originality of DCGD, please refer to \"Novelty and Originality of DCGD\" in our global response.\n\n>**Response to Q3.**\n\nThank you for raising this insightful question. While our proposed DCGD framework does share similarities with loss reweighting strategies in that it adaptively adjusts the weight of each loss term (in fact, $\\nabla_t \\mathcal{L}_{\\|\\nabla \\mathcal{L}_r^{\\perp}}$,\n\n$\\nabla_t \\mathcal{L}_{\\|\\nabla \\mathcal{L}_b^{\\perp}}$ in DCGD) at each iteration, it is more accurately described as a gradient surgery approach. This is because DCGD focuses on manipulating the direction of the updated gradient to address gradient conflicts and dominating gradient issues. As discussed in Appendix B, various multi-task learning algorithms [1,2,3] based on gradient surgery techniques can be seen as special cases of DCGD.\n\nAdditionally, since loss reweighting schemes operate fundamentally differently, combining DCGD with loss reweighting strategies could offer potential synergies. For example, as shown in Section 5.3, integrating DCGD with existing loss reweighting methods like LRA and NTK leads to improved performance.\n\n[1] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2020). Gradient surgery for multi-task learning. *Advances in Neural Information Processing Systems*, *33*, 5824-5836.\n\n[2] Liu, B., Liu, X., Jin, X., Stone, P., & Liu, Q. (2021). Conflict-averse gradient descent for multi-task learning. *Advances in Neural Information Processing Systems*, *34*, 18878-18890.\n\n[3 ]Désidéri, J. A. (2012). Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. *Comptes Rendus Mathematique*, *350*(5-6), 313-318.\n\n>**Response to Q4.**\n\nPlease refer to “more than two losses” in our global response. \n\n>**Response to Q5.**\n\nThank you for the interesting suggestion and for bringing the references [4,5,6] to our attention. We have carefully studied these papers and recognize that measuring gradients in function space rather than parameter space offers valuable insights for optimizing PINNs. While more in-depth experimental validation is needed to fully understand gradient conflicts in function space, we have explored this approach by measuring angles using the natural gradient, as illustrated in **Figure 2** of the attached PDF. Our findings indicate that gradient conflicts occur in function space similarly to parameter space. Furthermore, our method appears to handle these conflicts effectively.\nWe believe that extending our framework to consider the dual cone in function space could be a promising direction for future research.\n\n[4] Müller, J., & Zeinhofer, M. (2023, July). Achieving high accuracy with PINNs via energy natural gradient descent. In *International Conference on Machine Learning* (pp. 25471-25485). PMLR.\n\n[5] Zeng, Q., Kothari, Y., Bryngelson, S. H., and Schaefer, F. T. Competitive physics informed networks. In The *Eleventh International Conference on Learning Representations.*\n\n[6] Müller, J., & Zeinhofer, M. Position: Optimization in SciML Should Employ the Function Space Geometry. In *Forty-first International Conference on Machine Learning.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Global response to all reviewers**

We would like to express our sincere appreciation for the insightful comments and valuable suggestions provided by all reviewers. We assure you that all comments and suggestions will be thoroughly addressed in the revised manuscript. In this global response, we will summarize and address the common questions and key comments raised by the reviewers.
>  **Novelty and originality of DCGD** (*in response to Reviewers Nfs2, SdDT*)

To the best of our knowledge, our work represents the first instance of applying the concept of dual cones specifically to gradient manipulation within the context of gradient descent. This proposed framework is novel not only for PINNs but also across the broader field of multi-objective optimization.

While dual cones are indeed a fundamental concept in optimization, particularly in relation to constrained convex optimization (e.g., duality theory and formulating optimality conditions for convex problems), as noted by Reviewer Nfs2, our approach introduces a novel application of dual cones to manage gradient conflicts. This innovative use in the context of gradient descent is an area that has not been previously explored.

We would like to highlight the key contributions of our work. Existing strategies to resolve gradient conflicts are often ad hoc and lack a systematic framework. In contrast, our method provides a principled approach by utilizing the dual cone characterization to avoid gradient conflicts, as detailed in Theorem 4.2 and Proposition 4.3. This approach allows DCGD to be viewed as a generalization of various multi-task learning (MTL) algorithms, as discussed in Appendix B. Consequently, the proposed method has the potential to advance the development of future multi-objective optimization algorithms.Furthermore, our convergence results, presented in Theorem 4.5, demonstrate that DCGD converges not only to stationary points but also to Pareto-stationary points in a multi-objective non-convex setting. This provides both theoretical and empirical advantages compared to methods such as SGD and Adam, which guarantee convergence only to stationary points.

> **Broader Implications and Potential Impact Beyond PINNs** (*in response to Reviewer Nfs2*)

Our initial focus was on PINNs, but as suggested by Reviewer Nfs2, DCGD can be applied to various modern machine learning problems where multiple loss functions need to be managed simultaneously, such as in Physics-Informed Neural Operators (PINO), multi-task learning (MTL), and machine unlearning (where both forgetting and retraining losses must be considered).

To demonstrate the extensibility and applicability of DCGD, we have conducted additional experiments applying DCGD to PINO (in response to reviewer Nfs2) and machine unlearning problems. As shown in **Table 1** in the attached PDF, DCGD improves performance compared to optimally tuned methods when applied to PINO. Furthermore, in the context of machine unlearning, DCGD enables unlearning to be performed without compromising the quality of the generated images, as illustrated in **Table 2** and **Figure 1** in the attached PDF file.

> **More than two losses** (*in response to Reviewers NFZX, SdDT*)

- **Easy and simple approach**: For cases involving more than two loss functions, such as those arising from multiple governing equations and boundary conditions in PINNs, DCGD can be effectively applied by treating the multiple losses as a combination of residual loss and the other losses. For instance, in Section 5.2, we applied this approach to A-PINNs with three loss functions and obtained improved results.
- **More general approach**: Let us consider the total loss function $\mathcal{L}(\theta)$, which is the sum of multiple loss terms:
$$ \mathcal{L}(\theta):= \sum^{n}_{i=1}\mathcal{L}_i (\theta),  \text{where }i=1,2, \cdots, n $$

Denote by $\mathbf{K}^*_t$ the set of vectors satisfying $\langle u, \nabla \mathcal{L}_i(\theta_t) \rangle \geq 0$ for all $i = 1, 2, \cdots, n$. Then we can characterize $\mathbf{G}_t$, which is a subset of $\mathbf{K}^*_t$, by similar approach developed in this paper. For example, let $\mathbf{G}^{ij}_t$ be a subset of a dual cone region for $\nabla \mathcal{L}_i(\theta)$, $\nabla \mathcal{L}_j(\theta)$ which defined as follow:



$$\mathbf{G}_t^{ij} := \big\lbrace c_1g_i+c_2g_j+c_3g_c^{ij} \big| c_1, c_2, c_3\geq 0  \big\rbrace$$

where 
$g_k = \nabla_t \mathcal{L}_{\|\nabla \mathcal{L}_k^\perp} $ for $k= i,j$ and $g_c^{ij}$ is a unit vector that is orthogonal to both vectors $g_i, g_j$.

We can get $g_c^{ij}$ by a cross product of  $g_i$ and $g_j$ (More specifically, after express each vector by sum of $\nabla \mathcal{L}_i (\theta), \nabla \mathcal{L}_j (\theta)$ ).

Then we can determine the subset of dual cone as follows: $\mathbf{G_t} := \mathop{\bigcap}\limits_{i\neq j}^{n} \mathbf{G}_t^{ij}$.  In other words, the principles developed in this paper allow for the characterization of the dual cone region even when dealing with more than two loss functions.

> **Computational cost** (*in response to Reviewers Nfs2, Bm5Z*)

We compared the training speed of the proposed method with other optimization algorithms in Appendix D.2 of the original manuscript. It is important to highlight that DCGD algorithms are essentially first-order (explicit) gradient methods. Therefore, while there may be differences in speed due to the complexity of the update rule, these differences in training speed are not significant.

Meta-review: This work introduces a novel approach for solving multi-objective optimization as arises in the training of physics-informed neural networks. The idea is simple and natural: to select an update direction that is guaranteed to be a descent direction for each component of the loss function individually. The authors achieve significant improvement in the resulting accuracy, on a wide range of PINN benchmarks. The approach is nonintrusive in terms of the other aspects of PINN training and might be equally useful for multiobjective training in other domains.

In summary, the method is well-motivated, has potential for broad applicability, and is thoroughly evaluated (at least in the domain of PINNs, which is the focus of this work) I therefore recommend acceptance.
Decision: Accept (poster)