ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks
Authors: Qiang Liu
Mengyu Chu
Nils Thuerey
Venue: ICLR 2025 Spotlight (2025)
Keywords: Physics Informed Neural Networks
Multi-task learning
Conflicting gradients
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Our method (1) mitigates the conflicts of sub-gradients during PINN training and (2) substantially reduces the computations.

Abstract: The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. We provide a mathematical proof showing the convergence of the ConFIG method, and it is evaluated across a range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG

Published as a conference paper at ICLR 2025
CONFIG: TOWARDS CONFLICT-FREE TRAINING OF PHYSICS INFORMED NEURAL NETWORKS
Qiang Liu', Mengyu Chu & Nils Thuerey'
School of Computation, Information and Technology Technical University of Munich Garching, DE 85748
SKL of General AI Peking University Beijing, CN 100871 mchu@pku. edu on
(qiang7.1iu, nils. wtw.etthueeeeeee @tum. de
ABSTRACT
The loss functions of many learning problems contain multiple additive terms that disagree and yield conflicting update directions. For Inysice-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equa- tions are particularly interesting as they are well-established highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specifii gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage mo- mentum to accelerate optimizations by alternating the back-propagation of differ- ent loss terms. We provide mathematical proof showing the convergence of the ConFIG method, and it is evaluated across range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to base- line methods. We also test the proposed method in a classic multi task benchmark, where the ConFIG method likewise exhibits highly promising performance. Source code is available at uthhtps::// //tp::://ummeee/ -pm.. github....... io/ConFIC
INTRODUCTION
Efficiently solving partial differential equations (PDEs) is crucial for various scientific fields such as fluid dynamics, electromagnetics, and financial mathematics. However, the nonlinear and high- dimensional PDEs often present significant challenges to the stability and convergence of traditional numerical methods. With the advent of deep learning, there is a strongly growing interest in using this technology to solve PDEs (Han et al. 2018 Beck et al. 2023 In this context, Physics Informed Neural Networks (PINNs) (Raissi et al. [ et al. and differentiable ansatz for the underlying physics.
2022 leverage networks continuous
PINNs auto-differentiation of coordinate-based neural networks, i.e., implicit neural represen- tation (INR), to approximate PDE derivatives. The residuals of the PDEs, along with boundary and
initial conditions, are treated as loss terms and penalized during training achieve physically plau-
sible solution. Despite their widespread use, training PINNs is a well-recognized challenge (Cuomo)
al. 2022 Lino et al. 2023, Wang al. [021] Krishnapriyan al. 2021 Wang et al. 2022) due to
due
several possible factors like unbalanced back- propagated gradients from numerical sti stiffness (Wang al., 2021), different convergence rates among loss terms (Wang et al. (022). PDE-based soft
constraints (Krishnapriyan et al. (02T), poor initialization (Wong et al. (2024). and suboptimal sam- pling strategies (Daw et al.) 2023). Traditional methods to improve PINN training typically involve adjusting the weights for PDE residuals and loss terms for initial/boundary conditions (Liu & Wang, 2021 McClenny & Braga-Neto 2023] Son et al. 2022). However, although these methods claim to have better solution accuracy, there currently consensus the optimal weighting strategy.
Meanwhile, methods manipulating the gradient of each loss term have become popular in Multi- Task Learning (MTL) and Continual Learning (CL) (Riemer et al., 2019 Farajtabar et al.) 2020 Yu et al. 2020) to address confficts between loss-specific gradients that induce negative transfer (Long
al. 2017 and catastrophic forgetting (Kirkpatrick al. 2017). Unlike weighting strategies that

---

Published as a conference paper at ICLR 2025
adjust the final update direction solely by modifying each loss-specific gradient's magnitude, these methods take greater flexibility and often alter the directions of loss specific gradients. We notice that similar conflicts between gradients also arise in the training of PINNs. First, the gradient mag- nitude of PDE residuals typically surpasses that of initial/boundary conditions (Wang et al. 2021 causing the final update gradient to lean heavily towards the residual term. Additionally, PDE resid- uals often have many local minima due to the infinite number of PDE solutions without prescribed initial/boundary conditions (Daw et al. 2023). Consequently, when the optimization process ap- proaches local minima of the PDE residual, the combined update conflicts with the gradient of initial/boundary conditions, severely impeding learning progress.
To illustrate gradient conflicts in PINNs. we show a toy case in Fig. where Ci has smaller gradients and L2 has larger gradi- ents but multiple minima. Training with L + XC2 using suboptimal À values (e.g., = 1) leads to gradient conflicts and the convergence
Adam
Adam ConFIG (ivsi
g2
gl + 92
C1 C2
91
CuaPPG
local minima of C2.
93
In the following, we propose the Conflict- Free Inverse Gradients (ConFIG) method to mitigate the conflicts of loss-specific gradients during PINNS training. Our ap- proach provides an update gradient 9ConFIG that reduces all loss terms prevent the optimization from being stuck in the local minima of specific loss term. With the ConFIG method, the optimization in Fig. | converges to the shared minimum for both losses. Our method is provably convergent, and characterized by the following properties:
91
Figure 1: Visualization of toy example showing the conflict between different losses during optimization.
The final update direction does not conflict with any loss-specific gradients. The projection length of the final gradient on each loss-specific gradient uniform, ensur- ing that all loss terms optimized at the same rate.
The length of the final gradient is adaptively scaled based on the degree of conflict between loss-specific gradients. This prevents the optimization from stalling in the local minima of any specific loss term in high-conflict scenarios.
In addition, we introduce a momentum-based approach expedite optimization. By leveraging momentum, we eliminate the need to compute all gradients via backpropagation at each training iteration. In contrast, we evaluate alternating loss-specific gradients, which significantly decreases the computational cost and leads to more accurate solutions within given computational budget.
2 RELATED WORK
Weighting Strategies for PINNs' Training Some intuitive weighting strategies come from the penalty method of constrained optimization problems, where higher weights are assigned to less- optimized losses (Liu & Wang. 2021 Son et al., 2023 McClenny & Braga-Neto (2023) extended this idea by directly setting weights to sample points with soft attention masks. On the contrary, Bischof & Kraus (022)) advocated equalizing the reduction rates of all loss terms guides train- ing toward Pareto optimal solutions from an MTL perspective. Meanwhile, |Wan et al. (202T) addressed numerical stiffness issues by determining the weight according the magnitude each loss-specific gradient. WWang et al. (2022) explored training procedures via the Neural Tangent Ker- (NTK) perspective, adaptively setting weights based on NTK eigenvalues. Xiang al. (2022) instead employed Gaussian probabilistic models for uncertainty quantification, estimating weights through maximum likelihood estimation. Others have pursued specialized approaches. E.g.. van der Meer et al. (2022) derived optimal weight parameters for specific problems and devised heuris- ties for general problems. de Wolff et al. (2022) use an evolutionary multi-on muutiiobjeective algorithm to find the trade-offs between individual loss terms in PINNs training. [Shin et al. (020) conducted convergence analyses for PINNs solving linear second-order elliptic and parabolic type PDEs, em-
ploying Lipschitz regularized loss. In contrast adaptive methods, Wight & Zhao
2021 proposed
2

---

Published as a conference paper at ICLR 2025
fixed weighting strategy for phase field models. They artificially elevated weights for the loss term associated with initial conditions.
Gradient Improvement Strategies in MTL and CL Gradient improvement strategies in MTL and CL focus understanding and resolving conflicts between loss-specific gradients. [Riemer iet al. (2019) and Du et al. (2018) proposed using the dot product (cosine similarity) of two gradient vectors to assess whether updates will conflict with each other. |Riemer et al. (2019) introduced an additional term in the loss to modify the gradient direction, maximizing the dot product of the gra- dients. Conversely Du et al. (2018) chose discard gradients of auxiliary tasks if they conflict with the main task. Another intuitive method to resolve conflicts is orthogonal projection. |Faajtabarr al. (2020) proposed the Orthogonal Gradient Descent (OGD) method for continual learning, pro- jecting gradients from new tasks onto subspace orthogonal to the previous task. Chaudhry et al.) (2020) proposed learning tasks in low-rank vector subspaces that are kept orthogonal to minimize interference. Yu et al. (2020) introduced the PCGrad method that projects a task's s gradient onto the orthogonal plane of other gradients. Liu et al. (202b)) designed the IMTL-G method to ensure that the final gradient has the same projection length on all other vectors. Dong et al. (2022) utilized the Singular Value Decomposition (SVD) on gradient vectors to obtain an orthogonal basis. Davaloy & Valera 2022) introduces method that jointly homogenizes gradient magnitudes and directions for MTL. Quinton & Rey] (2024) propose a Jacobian descent method with an aggregation step, which constrains the update gradient into the positive cone in the parameter space avoid conflicts during training. Few studies in the PINN community have utilized gradient-based improvement strategies. Zhou al. (2023) introduced the PCGrad method PINN training for reliability assessment of multi-state systems. Yao et al. (2023) developed MultiAdam method where they apply Adam op- timizer for each loss term separately. Concurrent work from Hwang & Lim (2024) proposed a dual cone gradient descent method to mitigate gradient conflict when training PINNs.
METHOD
3.1 CONFLICT-FREE INVERSE GRADIENTS METHIO
Generically, we consider an optimization procedure with a set of individual loss functions, i.e.. {Li, fm Let [g), 92, ... Im denote the individual gradients corresponding to each of the loss functions. A grsdeett step with gradient g. will conflict with the decrease of C, if g. 9: is negative (Riemer et al. 2019 Du et al. 2018). Thus, to ensure that all losses are decreasing simultaneously along 9c. all m components of 92. 92. : 9m] ge should be positive. This condition is fulfilled by setting ge 92....... 9m w. where w = [W1, W2, .... W.. is vector with m positive components and M is the pseudoinverse of the transposed matrix M
Although a positive w vector guarantees a conflict-free update direction for all losses, the specific value of w, further influences the exact direction of ge. To facilitate determining w. we reformulate 9- as g- k (l) (() .... [gm)) - w. where ld(g))
g:/ //19.1 + E) is a normalization operator and k= > 0. Now, k controls the length of g.. and the ratio of ih's 's components corresponds
to the ratio of ga projections onto each loss-specific gi. i.e., [g.(S.(g.8 gi), where Se 9j) =
g. g: g | + e) is the operator for cosine similarity:
(. 9:) Se 9:) Solge, k((g:) 9g
[1:)) ge i; Sg gj) S.(g., kl((g))
Vi.j [1. (1)
[k2(gg)) 9. wj
We call v the direction weight. The projection length of ge on each loss- opss-specific gradient serves as an effective "learning rate" for each loss. Here, we choose w; = úū Vi,j E [1, ml to ensure uniform decrease rate of all losses, as it was shown to yield a weak form of Pareto optimality for multi-task learning (Bischof & Kraus 2021).
Meanwhile, we introduce an adaptive strategy for the length of g, rather than directly setting a fixed value of k. We notice that the length of g. should increase when all loss-specific gradients point nearly in the same direction since it indicates a favorable direction for optimization. Conversely, when loss-specific gradients We realize this by rescaling the length of g. to the sum of the projection lengths of each loss- specifcc gradient on it, i.e lg.l = _m Ig. (S
close to opposing each other, the magnitude of ge should decrease.
3

---

Published as a conference paper at ICLR 2025
The procedures above summarized in the Conflict-Free Inverse Gradients (ConFIG) operator G and we correspondingly denote the final update gradient 9c with IConFIG:
9ConFIG = G(gi .9m)
Σ
9, 9u
gu
9N 4 [[((g]) (())) _u(gm))
1 m
(3)
Here, 1m is unit vector with m components. mathematical proof of ConFIG's convergence in convex and non-convex landscapes can be found in Appendix [A.]] The ConFIG method utilizes the pseudoinverse of the gradient matrix to obtain a conflict-free direction. In Appendix AA.3 we prove that such inverse operation is always feasible long the dimension of parameter space larger than the number of losses. Besides, while calculating the pseudoinverse numerically could involve additional computational cost, it is not significant compared to the cost of back- poek-propagttion for each loss term. detailed breakdown of the computational cost can be found Appendix A.6
3.2 TWo TERM LOSSES AND POSITIONING W.R.T. EXISTING APPROACHES
For the special case of only two loss terms, there is an equivalent form of ConFIG that does not require a pseudoinverse:
(( = (g) 9. + 92 go)g. 9. = 4 - + +((002..9))))
(4)
(5)
where Olgi,9 92)
92
gi g2
191
91 is the orthogonality operator. It returns a vector orthogonal to
from the plane spanned by 9I and 92- The proof of equivalence is shown in Appendix A.4
PCGrad (Yu et al. 2020) and IMTL-G (Liu al. 2021b) methods from multi-task learning studies also have similar simplified form for the two-loss scenario. PCGrad projects loss-specific gradients onto the normal plane others if their cosine similarity is negative, while IMTL-G rescales loss- specific gradients to equalize the final gradient's 's projection length each loss-specific gradient, as illustrated in Fig. 2a and 2b Our ConFIG method Fig. [cc employs orthogonal components of each loss-specific gradient, akin to PCGrad. Mcanwhile, it also ensures the same decrease rate of all losses, similar to IMTLL-G As a result, these three methods share an identical update direction but different update magnitude in the two-loss scenario. This provides valuable opportunity to evaluate our adaptive magnitude strategy: for two losses, the PCGrad and IMTL-G methods can be viewed ConFIG variants with different magnitude rescaling strategies.
The above similarity between these three methods only holds for the two-loss scenario. With more losses involved, the differences between them are more evident. In fact, the inverse operation in Eq. 3 makes the ConFIG approach the only method that maintains conflict-free free direction when the number of loss terms exceeds two. Detailed discussion can be found in A.2
3.3 CONFIG WITH MOMENTUM ACCELERATION
Gradient-based methods like PCGrad, IMTL-G, and the proposed ConFIG method require separate backpropagation steps to compute the gradient for each loss term. contrast, conventional weight- ing strategies only require single backpropagation for the total loss. Thus, gradient-based methods
gPOGrd
CConFia
(gL. 92)
OMTTL-G
Ogr: 91)
u(O
u O
029
(a)
9t
(b)
oigi
(C
g1
Figure 2: Sketch of PCGrad (a), IMTL-G (b), and our ConFIG (c) method with two loss terms. The PCGrad method directly sums two orthogonal components, and the IMTL-G method rescales the two vectors to the same magnitude. Our ConFIG method sums the unit vector of the orthogonal components and adjusts its magnitude with the projection length of each loss-specific gradient.
4

---

Published as a conference paper at ICLR 2025
Algorithm M-ConFIG
Require: 8o (network weights), (learning rate), 31, S2, (Adam coefficient), mo (Pscudo first momentum), mg mg.. mmgm. ol < [0. 0, ... (First momentum), <- 0 (Second momentum),
19 g22"" .. .mm] f 00,0, S....... 0), All operations on vectors are element-wise except G.
for 4 to do
= 1% +
fg: Lg: +
fg:: B1 f.ma: mmgm
(1 VA) Ci
Update the first momentum of g: Bias corrections for D first momentum terms
Mag.
(0l
1222.102
mmm fom.
1-3.00
1-
1-8 8 mm
my, - G(rigg figg. fmggm
ConFIG update of momentums Obtain the estimated gradient b. Update the pscudo first momentum Update the second momentum > Bias correction for the second momentum
9. m,
31 8mm=-1
/(1 31)
m, 8 + 0.
V: + 3201-1 - -(1 32)ge
+ 01((1= - 52)
8. 0-11 7mg (v
b Update weights of the neural network
end for
usually = ToC.)/ ( L,) times more computationally expensive than weighting methods, where To is the computational cost of backpropagation. To address this issue, we intro- duce an accelerated momentum-base variant of ConFIG: M- M-ConFIG. Our core idea is to leverage the momentum of the gradient for the ConFIG operation and update momentum in an alternating fashion to avoid backpropagating all losses in single step. In each iteration, only single momen- tum is updated with its corresponding gradient, while the others are carried over from previous steps. This reduces the computational cost of the M-ConFIG method 1/m of the ConFIG method.
Algorithm [ details the entire procedure of M-ConFIG. It aligns with the fundamental principles of the Adam algorithm, where the first momentum averages the local gradient, and the second momen- tum adjusts the magnitude of each parameter. In this approach, we calculate only a single second momentum using an estimated gradient based on the output of the ConFIG operation. An alterna- tive could involve calculating the second momentum for every loss term, similar to the Adam method (ao a 2023). However, we found this strategy to be inaccurate and numerically unstable compared to Algorithm! A detailed discussion and comparison can be found in the Appendix [
Surprisingly, M-ConFIG not only catches up with the training speed regular weighting strategies but typically yields an even lower average computational cost per iteration. This stems from the fact that backpropagating a sub-loss L; is usually faster than backpropagating the total loss E. L. Thus, is usually smaller than and r r/mm- < 1. This is especially obvious for PINN training, where a reduced number of sampling points are used for boundary/initial terms and are faster to evaluate than the residual term. In experiments, observed average value of r 1.67 and speed-up of 1.67/3 x 0.56 per iteration for PINNs trained with three losses using M-ConFIG.
4 EXPERIMENTS
In this section, we employ the proposed methods for training PINNs on several challenging PDEs. We also explore the application our method on classical Multi- Task Learning (MTL) benchmark
-3.93-1.89 0.16 2.20
2,620
-2.332
1.449
Prediction
NL
0.07 3.0e-10
8.3e-07
-9.75e-03
8.18e-03
SB
Burgers
Schnödinger
Kovaszaay
Belrrmi
Figure 3: Examples of PINN predictions and squared error (SE) distributions the test PDEs.
5

---

Published as a conference paper at ICLR 2025
outlook. Unless mentioned otherwise, every result is computed via averaging three training runs initialized with different random seeds, each using the model with the best test performance during the training. For detailed numerical values and standard deviations of the result in each ex- periment, please refer to the Appendix an9 10 Training configurations and paper-parameters of each experiment can be found in the Appendix A.12 An additional ablation study training hyperparameters can be found in the Appendix 13 shows that the gains from ConFIG are robust w.r.t. hyperparameter changes.
4.1 PHYSICS INFORMED NEURAL NETWORKS
Preliminaries of PINNs. Consider the initial boundary value problem of a general PDE for scalar function u(æ t): IR++1 - given by
Maa( (), x = N(u( (), xz, 22, 4 fax 0, S2 E (0,7),
(6)
a. Blu( x), = a), + 0, 80, E (0,7),
(7)
L[u( 0). xz, 0] = u(x k(x, O, E
(8)
Here, C. Hd is a spatial domain with 02 and Z denotes its boundary and closure, respectively. N and B are spatial-temporal differential operators, and f. 9 and h are source functions. To solve this initial boundary value problem, PINNs introduce a neural network a( t, 0) for the target function u(x, t), where 8 is the weights of the neural networks. Then, spatial and temporal dif- ferential operators can be calculated efficiently with mnto-differentiiiion of N [ā t, 0). x, t) and B (1(x 1, 0), æx f]. Solving Eq. 8 turns to the training of neural networks with loss function of
(( Maa 4, 0), x. 4.
B (1ú 14 0), F2s 44) +
[ū( 0,0), x,, O).
(9)
EN
Cx
where (z),tt) are the spatial-temporal coordinates for the data samples in the Rd+1 domain, Cv. LB and LI are the loss functions for the PDE residual, spatial and initial boundaries, respectively. These three loss terms give us three corresponding gradients for optimization: gv- 9s. and gr. In the experiments, we consider four cases with three different PDEs: 1D unsteady Burgers equation, 1D unsteady Schrödinger equation, 2D Kovasznay flow (Navier-stoker equations), and 3D unsteady Beltrami flow (Navier-stokes equations). Fig. 3 shows examples of the solution domain of each PDEs. A more detailed illustration and discussion of the PDEs and corresponding solution domains be found in Appendix A.7
and A.8 respectively.
Focusing on two loss terms. As detailed above, the similarity between our ConFIG method and the PCGrad/IMTL-G methods the two-loss scenario offers valuable opportunity to evaluate the proposed strategy for adapting the magnitude of the update. Therefore, we begin with a two-loss scenario where we introduce a new composite gradient, LgI, which aggregates the contributions from both the boundary (Cg) and initial (Lz) conditions. This setup also provides better insights into the conflicting updates between PDE residuals and other loss terms.
Fig. compares the performance of PINNs trained with our ConFIG method and other existing approaches. Specifically, we compare to PCGrad Yu et al., 2020 and the IMTL-G method (Liu
100
50
ConFIGgouns)
RI(%)
60
M-ConFIG
PCGrad
40-
IMTL-G G
20
MinMax
ReLoBRaLo
10a
10a
LRA
Burgers
Schrodinger
Kovasznay
Beltrami
Figure 4: Relative improvements of PINNs trained with two loss terms using different methods.

---

Published as a conference paper at ICLR 2025
Adam
Cx
ConFIG Ex
Adam Lez
ConFIG Cez
Burgers
Schrödinger
Kovasznay
Beltrami
10-2
10-3
10
10
10-1.
10-9
10-0.
10-+
201000
0
1000000
1000000
1000000
Epoch
Epoch
Epach
Epoch
Figure 5: Training losses of PINNS trained with Adam baselines and ConFIG using two loss terms.
100
80
ConFIGrour))
RI(%)
60
M-ConFIG
PCGrad
40
IMTL-G
20
MinMax
ReLoBRaLo
10a
10
LRA
Burgers
Schrodinger
Beltrami
Figure Relative improvements of PINNs trained with three loss terms using different methods.
ctt al., 2021b) from MTL studics as well as the LRA method (Wang et al. 2021), MinMax (Liu
& Wang 202T), and ReLoBRaLo (Bischof & Kraus 202T) as established weighting methods for PINNs. The accuracy metric the MSE between the predictions and the ground truth value on the new data points sampled in the computational domain that differ from the training data points. As the central question for all these methods is how much improvement they yield in comparison to the standard training configuration with Adam, we show the relative improvement (in percent) over the Adam baseline. Thus, +50 means half, -100 twice the error of Adam, respectively. (Absolute metries are given in the Appendix A.9
The findings reveal that only our ConFIG and PCGrad consistently outperform the Adam baseline. In addition, the ConFIG method always performs better than PCGrad. While the IMTL.- method performs slightly better than our methods in the Kovasznay and Beltrami flow cases, it performs worse than the Adam baseline in the Schrödinger case. Fig. [ compares the training loss of our ConFIG method with the Adam baseline approach. The results indicate that ConFIG successfully mitigates the training bias towards the PDE residual term. achieves an improved overall test performance by significantly decreasing the boundary/initial loss while sacrificing the PDE residual loss slightly. This indicates that ConFIG succeeds in finding one of the many minima of the residual loss that better adheres to the boundary conditions, i.e., it finds better overall solution for the PDE.
Adam
Lx
ConFIG (Ours) Ce
PCGrad Cov
Adam
Cx
ConFIG (Ours) Cx
PCGrad Cy
Adam
Es
ConFIG (Ours) Cr
PCGrad Cs
10 2
Burgers
Schrödinger
Beltrami
10
10
4
10-4
10
10-
10-6
20000
0
100000
0
100000
Epoch
Epoch
Epoch
Figure 7: Training losses of the Adam baseline, PCGrad, and ConFIG with three loss terms.
7

---

Published as a conference paper at ICLR 2025
Equaloee
MieMaax
RoLoBRaLo
URA
Equal
MinMax
RoloBRaLo
LRA
901
500
10
RIN
-10-
(a)
-11
Burgers
Kaseuniey
(b)
Burgers
Schnodager
Balrenii
Figure 8: Relative improvements of the ConFIG method with different direction weights, (a) Two- loss scenario. (b) Three-loss scenario.
80
ConFilGuuro)
M-ConFIGgpuni
60
PCGrad
I
23
MinMax
-107
ReLuBRaLo
-10%
LRA
(a)
Burgers
Schrodinger
Kovasanay
100
80
ConFIGrou-ii
M-ConFIG
PCGrad
IMTL-G
20
MinMax
-10
ReLoBRaLa
LRA
(b)
Burgers
Schrodingee
Beltrami
Figure 9: Relative improvements of the PINNs trained with different methods with the same wall
time. (a) Two-loss scenario. (b) Three-loss scenario.
Scenarios with three loss terms. Fig. compares the performance for all three loss terms. While the general trend persists, with ConFIG and PCGrad outperforming the other methods, these cases highlight interesting differences between these two methods. As PCGrad performs better for the Burgers and Schrödinger case, while ConFIG is better for the Beltrami flow, analyze the training losses of both methods in comparison to Adam Fig. 7 In the Burgers and Schrödinger equa- tion experiment, ConFIG notably reduces the loss associated with initial conditions while the loss of boundary conditions exhibits negligible change. Similarly, for PCGrad, minimal changes are observed in boundary conditions. Furthermore, owing PCGrad's bias towards optimizing in the direction of larger gradient magnitudes, i.e., the direction of the PDE residual, the reduction in its PDE residual term is modest, leading a comparatively slower decrease in the loss of initial condi- tions compared to ConFIG. In the case of the Beltrami flow, our ConFIG method effectively reduces both boundary and initial losses, whereas the PCGrad method slightly improves boundary and ini- tial losses. These results underscore the intricacies of PINN training, where PDE residual terms also play pivotal role determining the final test performance. In scenarios where the PDE resid- ual does not significantly conflict with one of the terms, an increase in the PDE residual ultimately reduces the benefits from the improvement on boundary/initial conditions, resulting better performance for PCGrad in the Burgers and Schrödinger scenario.
Adjusting direction weights. weights i6 1m to ensure a uniform decrease rate across all loss terms. In the following experi- ments, we use the different weighting methods discussed above to calculate the components of tt and compare the results with our ConFiG method. As demonstrated in Fig. 8 only our default strat- egy (equal weights) and the ReLoBRaLo weights get better performance than the Adam baseline. Moreover, equal setup consistently outperforms the ReLoBRaLo method, except for slight inferiority in the Kovasznay flow scenario with two losses. These results further validate that the equal weighting strategy is good choice.
our ConFIG method, we set equal components for the direction
8

---

Published as a conference paper at ICLR 2025
Evaluation of Runtime Performance While the evaluations in Fig. and have focused on accuracy after a given number of training epochs, potentially more important aspect for practical applications is the accuracy per runtime. This is where the M-ConFIG method can show its full potential, as its slight approximations of the update direction come with a substantial reduction in terms of computational resources (we quantify the gains per iteration in A.6)). Fig. compares the test MSE for M-ConFIG and other methods with constant budget in terms of wall-clock time. Here, M-ConFIG outperforms all other methods, even PCGrad and the regular ConFIG method which yielded better per-iteration accuracy above. This is apparent for all cases under consideration.
To shed more light on its behavior, Fig. 10 shows the test MSE of Adam, ConFIG, and M- ConFig for the most challenging scenario, the
Adam
ConFIG
ConFIG
3D unsteady Beltrami flow, with an extended training run. It shows that the advantage of M-
10-1
MSE
Epoch=19/213
ConFIG does not just stem from quick late or early decrease but rather is the result of a con- sistently improved convergence throughout the full training. This graph additionally highlights that the regular ConFIG method, i.e., without momentum, still outperforms Adam despite its higher computational cost for each iteration.
Epoch=171026
10-4
Epoch~00000
8.864
2000
4000
G000
8000
10000
faoll!
Figure 10: Test MSE of Adam baseline, ConFIG,
and M-ConFIG as functions of wall time.
Additional Tests on Challenging PDEs Re-
cently, several benchmark problems have been proposed as challenging tasks for training PINNs (Hao et al. 2023). We have also tested our ConFIG and M-ConFIG methods together with other methods on these challenging tasks. The results are summarized in Appendix. [A.I1] They show that, although our methods do not resolve all inherent difficulties of PINN training, our methods still consistently outperform the other methods. These results shows the general appeal of the proposed methods for challenging, and high-dimensional training scenarios.
4.2 MULTI-TASK LEARNING
While PINNs represent highly challenging and relevant case, we also evaluate our ConFIG method for traditional Multi-Tass Learning (MTL) as an outlook. We employ the widely studied CelebA dataset (Liu et al.) 2015), comprising 200,000 face images annotated with 40 facial binary attributes, making it suitable for MTL with m 40 loss terms. Unlike PINNs, where each sub-task (loss term) carries distinct significance, and the final test loss may not reflect individual subtask performance, the CelebA experiment allows us to use the same metric to evaluate the performance for all tasks.
We compared the performance of our ConFIG method with ten popular MTL baselines. Besides PCGrad and IMTL-- from before, we also compare to Linear scalarization baseline (LS) (Caruana 1997). Uncertainty Weighting (UW) (Cipolla al. 2018), Dynamic Weight Average (DWA) (Liu ct al. 2019), Gradient Sign Dropout (GradDrop) (Chen et al. 2020)) Conflict-Averss Gradient De- scent (CAGrad) (Liu et al. 2021a), Random Loss Weighting (RLW) (Lin at. (022). Nash bargain- ing solution for MTL (Nash-MTLL (Navon et al.; 2022). and Fast Adaptive Multitask Optimization (FAMO) (Liu al.) 2203).. We use two metrics to measure the performance of different methods: the
mean rank metric (, lower ranks being better) (Navon et al. 2022] Liu et al. (023)). represents the average rank of across all tasks. An M R of means consistently outperforming all others. In
addition, we consider the average F1 score (F1, larger is better) to measure the average performance on all tasks.
Fig. presents a partial summary of the results. Our ConFIG method or M-ConFIG method emerges as the oop-perormmnng method for both MR and Fj metric. Unlike the PINN study, we update 30 momentum variables rather than a single momentum variable for the M-ConFIG method, as indicated by the subscript '30.' This adjustment is necessary because a single update step is insufficient to achieve satisfactory results in the challenging MTL experiments involving 40 tasks.
Varying Number of Tasks Fig. 12 illustrates how the performance of the M-ConFIG method varies with different numbers of tasks and momentum-updiat steps in the CelebA MTL experi- ments. The performance of M-ConFIG tends to degrade as the number of tasks increases. This

---

Published as a conference paper at ICLR 2025
0.70
ConFIG
PCGrad
M-ConfiG 30
Grop
0.68
IMTL-G
CAGrad
NASH-MTL
0.00
FAMO
RLW
UW
MRL
Figure 11: Test evaluation terms of F1 and MR for the CelebA experiments. ConFIG and M- ConFIG 30 yield the best performance for both metrics, each of them favoring different metric.
1.00
1.00
0.75
-0.3
0,75
0,50
0.50
6:25
0.1
0.25
0.25
6.00
5
10
20
30
40
0.0
0.00
1
5
10
20
30
40
0.00
Figure 12:
The relative F1 value and training time of the M-ConFIG method with a varying number of tasks and gradient update steps in the CelebA experiment.
decline is attributed to longer intervals between momentum updates for each gradient, making difficult for the momentum accurately capture changes in the gradient. When the number of tasks becomes sufficienlly large, this performance loss outweighs the benefits of M-ConFIG's faster train- ing speed. In our experiments, when the number of tasks reaches 10, the Fi score of the ConFIG method (0.635) already surpasses that of M-ConFIG (0.536) with the same training time. However, increasing the number of gradient update can effectively mitigate this performance degradation. In the standard 40-task MTL scenario, increasing the number of gradient update steps per iteration 20 allows M-ConFIG to achieve performance level comparable ConFIG while reducing the training time to only 56% of that required by ConFIG. Notably, alternating momentum updates can sometimes even outperform the ConFIG method with the same number of training epochs, as seen in the uppdaee a 30 case in the MTL experiment, as well as in the Burgers and Kovasznay cases in the two-loss PINNs experiment.
Limitations A potential limitation of the proposed method is the potential performance degrada- tion of the M-ConFIG method as the number of loss terms increases. Additionally, the memory consumption required to store the momentum for each gradient grows with both the size of the pa- rameter space and the number of loss terms. As discussed in the MTL experiment, one approach to mitigate the performance degradation is update not just single gradient but a stochastic subset of gradients during each iteration, albeit at the cost of an increased computational budget. Nonetheless, further investigation and improvement of the M-ConFIG method's 's performance in scenarios with large number of loss terms is a promising direction for future research, aiming to broaden the appli- cability of the ConFIG approach. Meanwhile, we also notice that there are still many challenges in the training of PINNs, e-g., chaotic problems, that can not be addressed by eliminating the conflicts between different loss terms during the training. Future efforts, like improved network structures and imposed causality, will be required to further improve PINN training.
5 CONCLUSIONS
In this study, we have presented ConFIG, method designed to alleviate training conflicts between loss terms with distinct behavior. The ConFIG method achieves conflict-free training by ensuring a positive dot product between the final gradient and each loss-specific gradient. Additionally, we introduce a momentum-base based approach, replacing the gradient with alternately updated momentum for highly efficient iterations. In our experiments the proposed methods have shown superior per- formance compared with a wide range of existing training strategies for PINNs. The config variants even outperform SOTA methods for challenging MTL scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Qiang Liu acknowledges the support from the China Scholarship Council (No.20220612000) ) for his Ph.D research at the Technical University of Munich.
REFERENCES
Chandrajit Bajaj, Luke McLennan, Timothy Andeen, and Avik Roy. Recipes for when physics fails: recovering robust learning of physics informed neural networks. Machine Learning: Science and Technology. 4(1):0151,3, feb 2023. doi: 10. 1088/2632-22555abb4100 URLjtttps://dx
doi doi. 1088/228200000
acb416
Christian Beck, Martin Hutzenthaler, Arnulf Jentzen, and Benno Kuckuck. An overview deep learning-basse based approximation methods for partial differential equations. Dis-
crete and Continuous Dynamical Systems B, 28(6):367-33766, 2023. ISSN 1531-3492. doi: 10.39934/dcdsb.22222333 URL http::://wwwt hwt.:a/amcceeeree..rrrrrrrirccee 639b367 1444211144441133ee
Rafael Bischof and Michael Kraus. Multi-objective loss balancing for physics-informed deep learn- ing, 2021.
R. Caruana. Machine Learning, chapter Multitask Learning. Springer, 1997. doi: 10.1023/A: 1007379606734.
Arslan Chaudhry, Nacemullah Khan, Puneet Dokania, and Philip Torr Continual learning in low- rank orthogonal subspaces. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9900-9911. Cur- ran Associates, Inc., 2020. http:////e.o......a.a...aaa/a//3
httpc//ppprfeees////epppp files/
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzsehmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gra- dient sign dropout. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2039-2050. Cur- ran Associates, Inc., 2020. URL /t/p/ttpss//pprooeddn..eeerrrerr
files/
Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 7482-7491, 2018. doi: 10. (109//CVPR.208800777
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informet informed neural networks: Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022. doi: 10.1007/ (040090221-01444
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Mitigating propagation failures in physics-informet neural networks using eetain-essamll--eebleee (13) sampling. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Taco de Wolff, Hugo Carrillo Lincopi, Luis Martí, and Nayat Sanchez-Pi. Mopinns: an evolutionary multi-objective approach to physics-informed neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO '22, pp. 228-231, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392686. 5520304.3529077 URL UtLt:ttss://dol wwt./rrg/g/o http://...111111252323 (11111155200443522222
doi: 10.1145/
Xin Dong, Ruize Wu, Chao Xiong, Hai Li, Lei Cheng, Yong He, Shiyou Qian, Jian Cao, and Linjian Mo. Gdod: Effective gradient descent using orthogonal decomposition for multi-task learning. In Proceedings of the 3 ACM International Conference on Information & Knowledge Management, CIKM '22, pp. 386-395, New York, NY, USA, 2022. Association for Computing doi: 10.1145/351 110551888883555333 URL https: //doi.
Machinery, ISBN 978 450392365.. arq/10.1145//2 808 3557833

---

Published as a conference paper at ICLR 2025
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, 2018.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Pro- ceedings of Machine Learning Research, pp. 3762 3773. PMLR, 26-28 Aug 2020. URL htttps://proceedingg htt.oceedddassm.ppppessssssaeeia http://vooooaaaaaaaaaaaaaaaaaaaa wtw.aaaaaaaaaaaaaa.a.. html
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- tional Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249 256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: htttps::/proceediimgg wtw.eeeeiiin.m.............. wttp..ppssss.s..seeese wttp://gooooooooooo.oo 0aa0aa.hhh.. htmmllll
Lars Grasedyck. Hierarchical singular value decomposition of tensors. SIAM Journal on Matrix Analysis and Applications, 31(4):2029-2054, 2010. doi: 10.1137/09074118
I. Gromeka. Some cases of incompressible fluid motion. Technical report, Kazan Federal University, 1881.
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high- nigh-dimensional partial differential equa- tions using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505- 8510, 2018. doi: 10. 10.1073/pnas. 1718942115.. URL htthh::////// https://www.pan. pwws. http://www..aaa..//////e//////// org/dos/abs/ 10.1073/ 10100///aa...... 17189428 8942115
Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, et al. Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes. arXiv preprint urXiv:2306.08827 2023.
Philipp Holl, Nils Thuerey, and Vladlen Koltun. Learning to control pdes with differentiable
physics. In International Conference on Learning Representations, 2020. URL https:
//openzevien ewweereview.... forss//ooummiitm wwworrmmddeyeeinnn5...
Youngsik Hwang and Young Lim. Dual cone gradient descent for training physics-informed neural networks. Advances in Neural Information Processing Systems, 2024.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In
International Conference on Learning Representations, 2022. URL httphtttss:///eeeeviiii
net/forum iiddiiddd==o T8w rnuGL
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A.
Demis Has- Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic for- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. getting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521- 3526, 2017. doi: 10. 00.1073/pna..661188
URL https:/ http:/wwppppssss. wrg//oi/a/a//o
http:/pps...11111111..1
I. I. G. Kovasznay. Laminar flow behind a two-dimensional grid. Mathematical Proceedings of the Cambridge Philosophical Society, 44(1):58-62, 1948. doi: 10.1017/00500041102399999
Aditi S. Krishmapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Baijiong Lin, Feiyang YE, Yu Zhang, and Ivor Tsang. Reasonable effectiivness of random weight- ing: A litmus test for multi-task learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL htt.ttss::// http::://pppevvveeeeeee net./ tttss//formmmidd=tFFF]
Mario Lino, Stathi Fotiadis, Anil A. Bharath, and Chris D. Cantwell. Current and emerging deep-learning methods for the simulation of fluid dynamics. Proceedings of the Royal So- ciery A: Mathematical, Physical and Engineering Sciences, 479(2275)2022305888 2023. doi: 10. 10.1098/sspa.2033.0088 URL /ttp:///e/////taaaoooooooooooaoa
10. 1 10100098ppppa wwt.apa..22.00....
12

---

Published as a conference paper at ICLR 2025
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34, 2021a.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization, 2023.
Dehao Liu and Yan Wang. A dual-dimer method for training physics-constrainee neural networks with minimax architecture. Neural Networks, 136: 1 12-125, 2021. ISSN 0893-6080. doi: https:
101. 12.028.
Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin I Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021b. URL
IMP IMXEWWVVV
S. Liu, E. Johns, and Davison. End-to-end multi-task learning with attention. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1871-1880, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society. doi: 10. 10.1109/ 2019 19.00197.
www.eeeepteeeccceeee...o URL http::///wo...eeeoooeeeeeceeeeos
CVPR. 2019 00090 97
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Philip S. Yu. Learning multiple tasks with multilinear relationship networks. Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp- 1593-1602, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 5185699669
Levi D. McClenny and Ulisses M. Braga-Neto. Self-adaptive physics-informed neural networks. Journal of Computational Physics, 474:111722, 2023. ISSN 0021-9991. doi: https://doi.rr//10
je.22.icp.202221172721
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint arXiv:200.000117, 2022.
Pierre Quinton and Valérian Rey. Jacobian descent for multi-objective optimization, 2024. URL https://arxix: or abs /20.0///225.222 6232
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-709, 707, 2019. ISSN 0021-9991 doi: https://doi. org/10. 101100..016/jjjpp.2200 8. 10.045. URL https: wwwp://www.... sci enwedineceettcc cwm/ sci ence/
Aaticle/pii//0000 9991 183091188312551
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish. Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing in- terference. In International Conference on Learning Representations, 2019. URL https:
//openzeview. www.eeewwee/foouumm.. Btt./ooommniigiigaaaaeaaeaa
Yeonjong Shin, Jérôme Darbon, and George Em Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. Commu- nications in Computational Physics, 2845):2022-2074, 2020. ISSN 1991-7120. doi: https:
Hwijae Son, Sung Woong Cho, and Hyung Ju Hwang. Enhanced physics-infommn neural networks with augmented lagrangian relaxation method (al-pinns). Neurocomputing. 548: 146:12 2023. ISSN 0925-2312. doi: neucom. 2023.1222222
Clifford Truesdell. The Kinematics of Vorticity. Indiana University Press, 1954.
Remco van der Meer, Comells W. Oosterlee, and Anastasia Borovykh. Optimally weighted loss functions for solving pdes with neural networks. Journal of Computaaionnl and Applied Mathe- matics, 405:1 13887, 2022. ISSN 0377-0427. doi: https//dbbi.or///..01/........ 20211 13887.
13

---

Published as a conference paper at ICLR 2025
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in physics-informa neural networks. SIAM Journal on Scientific Computing, 43(5): A3081, 2021. doi: 10.1 10.1137/20M1318044
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 10768, 2022. ISSN 0021-9991. doi:
https://doi.or//10 1ttp//0.11011. 1.100.6611...... jtpp://...
Colby L. Wight and Jia Zhao. Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks. Communications in Computational Physics, 29(3):33-954,, 2021. ISSN 1991-7120. doi: https://doi.or//11.422222
0A-2020-0086.
Jian Cheng Wong, Chin Chun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions Artificial Intelligence, (3):985-1000, 2024. doi: 10. 00.10999A112222221222322
Zixue Xiang, Wei Peng, Xu Liu, and Wen Yao. Self-adaptive loss balanced physics-informed neural networks. Neurocomputing, 496:11-34 2022. ISSN 0925-2312. doi: https://doi.org http:///oooor//... 101
neucom.2022.05.015.
Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. Multiadam: parameter-wise scale-invariant optimizer for multiscale training of physics-informed neural net- works. In Proceedings of the 40th International Conference Machine Learning, ICML'23. JMLR.org, 2023.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5824-5836. Curran Associates, Inc., 2020. URL http:://prrceccinnggg..rrr.....s wwtpppppwpppppppppppppppppppepee wttperfffilesepppeeeeeee /ttp SiE22aaaeeetaadaeee3333333322222 pap
pdf
Taotao Zhou, Xiaoge Zhang, Enrique Lopez Droguett, and Ali Mosleh. A generic physics-informed neural network-based framework for reliability assessment of multi-state systems. Reliability En-
gineering & System Safery, 229:108835, 2023. ISSN 0951-8320. doi: http:///do..rg//0.10166
2022 URL https: www. sci iiccencceiiic rect com/ htmm//iienc//rrtcllee
p11/80051833022044333
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 CONVERGENCE PROOF FOR THE CONFIG METHOD
In this section, will give the proof for convergence for both convex (Theorem 1) and non-convex landscapes (Theorem 2) for our ConFIG method.
Theorem 1. Assume that (a) objectives Lı; L2..... ...m are convex and differentiable; (b)The gradient = VeL = Di=l VeL, is Lipschitz continuous with constant I, > O. Then, update
along ConFIG direction 9ConFIG with step size S E will converge to either a location where ConFrG or the optimal solution.
Proof. A Lipschitz continuuus with constant I, >> 0 gives negative semi-definite matrix V LT. we quadratic expansion of around [(0), we will get
C ((0)) S (()) VC(O)" (0t 0) + V2 ((0) 0+ 0f²
(10)
(()) + g" (0+ 0) + 1
0 0|2.
The 0+ is obtained through an update as 0- 0 79ConFIG resulting in
(0+) s L()) ~'"" gConFTG + T !gcomric! 2
(11)
1 g CCofFIG + L |9ConFiG|
[()) 7
Our @ConFIG has following features:
An equal and positive projection on each loss-specific gradient:
g. 9cwero
9. grsemm
Ig growing
9. [graam
0, i.e., (e(gs, [CoaFIG Se 9ConFIG) S. 0;
A magnitude equals to the sum of projection length of each loss-specific gradient on it: 9ConFiG Σm 9, ((gcoatta) = Σ gi Sc.
Given these two conditions, we have
m
m
9, DConFIG
gg gConFIG)
(1gs | Sc)
(12)
[CoFFIG
(19:1S.)
[ConFiG
turning 1 11 into
1 C (0+) < L(0) Aggcouri 2 +
2 L |
(13)
If & 2. we have } I Igcuarial -/9ccaarcc < O, which finally gives L ( < L (0). Note that when y S 3. the equality holds and only holds when |gConFial = 0 where conflict direction doesn't t exist and optimization along any direction will results at least one the losses increases.
differentiable and possibly non- convex; (b)The gradient g = VoL Σm, VoL; is Lipschitz continuous with constant > 0.
Theorem 2. Assume that (a) objectives Li, Lz... f.n
Then, update along ConFIG direction 9CoFFIG with step size y S 2 will converge to either a location where [ or the stationary point.
15

---

Published as a conference paper at ICLR 2025
Proof. Using the descent lemma, update on ICoaFIG with step size y $ gives
C ( s c. (0*) 7 lgcarta
(14)
where the superscript is the index of optimization iterations.
Due to [ConFIG = Σ m Σ m
1=1 e9 igl
= m gil and the triangle inequality where Σ Igi 2
1= 9sl. we have
C (0++1)
S S (0%)
CCOFIG 1
2
= =
g sk.
i=1
(15)
m
S $0(00)
k S
(0*)
2
9 S2.
Summing this inequality over 1,2,
K results in
K
K
C (0h)
L (B&+1)
Σ
9
(16)
Sk
k=
By defining m (S%)
0, we have
Σ
ا < 70
Ic (00) - (ox+1)].
(17)
k=1
Finally, averaging Eq 17 on each side leads to
2 ya²K (00) L (OKK1)]]
min ISKSK
gg
1
2
(18)
As + co, either the |gCooFIG| is obtained and the optimization stops. or the last term in the inequality goes 0, implying that the minimal gradient norm also goes to zero, i.c., a stationary point.
A.2 COMPARISON BETWEEN CONFIG, PCGRAD AND IMTL-G
Algorithm 2 PCGrad method
Require: 0 (network weights), [L, L2, Lm] (loss functions)
[9) 92. 9m
VoLi. VoL2, 7
[9. J2.
9m
91, D2. ... gm
for i + to do
uni formly
for j
11 2, ml in random order do
if / j &
9. 9,
< 0 then
9, g 9i)
end if
end for OPOGrd
Σl=
end for
PCGrad method. Algorithm 2 outlines the complete algorithm of the PCGrad method. The core idea is to use O(gz. 91) replace 91 when 91 conflicts with gu i.c., gi g2 0. Since Ogg2. ) 92 = 0, the conflict is resolved. In the simple case of two loss terms, this results in the final gradient gecaod O(gi.g2) 92) O(zz. 91). This final update gradient does not conflict with
16

---

Published as a conference paper at ICLR 2025
all loss-specific gradients, as [O(gi. 92) Ogg,, g1)l 91 = O(g2.9) "ia"gg"" 91 =911/
[g] ga)
1g lo. q2l
O.
l
However, the PCGrad method can not guarantee a conflict-free update when there are more loss- specific gradients. To illustrate this, let us consider the case of three vectors, g1 g2, 93. where each vector conflicts with the other. According to the PCGrad method, we first replace gi with (g2, 91); i... 91 Ogg 91) that gi 92 0. If the updated 91 still conflict with gs. we will then further modify 91 as 91 (gs. 91) However, now we can only guarantee that 91 0, but we can not guarantee that 9 still not conflicting with 92-
The PCGrad method introduces random selection to mitigate this drawback, shown in Al- gorithm 2 but this does not fundamentally solve the problem. Fig. [3 illustrate simple case for the failure of the PCGrad method where 9i [1.0, 0, 0.1], 92 [-0.5, v3/2, 0.1],
and 93 = [00.5, - 3//2,, 0.1). The final update vector for the PCGrad method is gPOGra 1-0.551, - -0.203, 0.658), which conflicts with Φi while the update direction of our ConFIG method
is ICoOFIG = [0, O, 0.3). which is not conflict with any loss-specific gradients.
PCGrad
ConFIG
0.5
1.0
1.0
1.0
-0.5
0.0
0.5
1,9-1.0
Figure 13: A simple failure model for PCGrad method.
Another feature of the PCGrad method is that the projection length of the final update gradient GrCora on each loss-specific gradient is not equal, ie., Ircompu * Ipc)m) and geeord is usually biased to the loss loss-specific gradient with higher magnitude. We can illustrate this point with two gradient examples:
91
M(C = (O(g), (2) + ((g2.1)))
[g1
91
O (T)
g1
91 92
91
(19)
91
02
92 2
9
[g] )2)
|gil
gg 10
1g: [1
(g 92)]
Similarly,
@PCGmal(gg2) = |g [1
S (g1. g=)]
(20)
Thus,
(
91
(21)
Drcamad {g)))
1g
17

---

Published as a conference paper at ICLR 2025
A similar conclusion also holds for the case with more loss loss-specific gradients. The essence here the magnitude of the orthogonal operator which is proportional to the magnitude of loss toss-specific gradients:
(g 9,)l
V(a
V (g)
9, 9j 19:13
g.)²
(19,1 12
(g, 9,)² 1g. 1
9: 9j
9,) 9
9,
(22)
(g] 9,)²
|,,1
1g:
V 1 (g
19
When summing all these orthogonal values together, the final gradient will be biased to the Ogi 9j) with a larger gradient. Meanwhile, since (gi, 9;) biased towards gj rather than (due to Selgi, P(g.,9)) = 0), the final gradient will favor to the loss-specific gradient with larger magni- tude.
IMTL-G method. In contrast to the PCGrad method the IMTL-G method is designed to guarantee an equal projection length, i.c.
9IMTL-G U(gg)) 9IMILGG U(g;)
(23)
To achieve this goal, the IMTL-G method rescales the magnitude of each loss-specific gradient with calculated weights:
gIMTL-G
ag
(24)
a22 03 ... amm] = 91-1U UT (D.UT)
Q1 = Σ
(25)
UT [(( - U(g ), (, - -U(g3) )
_((TT) - [(ghT)]
D 19 - .g. 9 ... 0 9m
For the two gradient scenario, this will give [ar, a2) = (1/))/((g)1 + (g))),1911 l/g]/ + [g2])]]
resulting in a same magnitude for two loss-specific gradients as (lg) 9 ( (9 + [gl)). The final gradient vector doesn't conffict with 91 and g2.
(IMLLOUU)())))
Ig |g2
91
92
T
91
1g
+ gg
91
92
[g
[g
|1
(26)
(1 + (g1.
1g1 + 1g2l
As mentioned in Appendix. 3.2 the direction of the final update gradient for IMTL-G and our ConFIG is the same when there are only two vectors, but their magnitudes differ. For the Adam optimizer, the gradient's absolute magnitude is unimportant since ultimately rescales the gradient elements. The crucial aspect is how the gradient changes in both direction and magnitude. When the gradient changes rapidly, Adam's corresponding learning rate will decrease, as discussed in Ap- pendix. AA.3] Since the direction of the final gradient in IMTL-G and ConFIG is the same, how their magnitudes change ultimately affects their performance with the Adam optimizer. The magni- tude of the final gradient with two gradients are 2 2 + (g 92)) /2( g1 (olll//l/(((g + (g2)) and V[1 + S.(g)), (.)) /2(lg) + [92l) for IMTL-G and ConFIG method, respectively. Thus,
|IMTLG
2(1g: 11 (11 122))
(27)
[ConFIG
91 1g2
18

---

Published as a conference paper at ICLR 2025
A significant difference arises when the magnitudes of 91 and 92 different. For example, if 91 >> 92. (19xl192))/((n 1g2l) ~ 92 while 91 + 92 ~ 91- This means that the magnitude of 9IMTLGG tracks the changes in the magnitude of the smaller vector, whereas ConFIG method tracks the magnitude of the larger vector. This causes the difference between IMTL-G and our ConFIG methods in two loss-term vector situations.
IMTL-G
ConFIG
0.5
0.0
0.5
0.5
-0.5
0.0
0.0
y
005
1.0
1.9
Figure 14: A simple failure model for IMTL-G method.
Another major difference between IMTL-G and our ConFIG methods occurs when more gradi- ent terms are involved. The coefficient a of IMTL-G method might have negative values since Eq. 25 only guarantees that 9IMTL-G ((g) = gINTL-G U(g;)) but not gIMTL-G U(gg)
JIMIL-G M( 2 0. It is easy to find such a situation where the negative coefficient will result in conflicting update. Fig. 14 illustrates such an example where 91 [0.0412, 0.4295, 0.9394], 92 = 00.3571, 0.5491, 14 and [0.9823, 0.9361, 0.0552). Although both our ConFIG and gIMTL-G (g;)) they are in exactly opposite direction. The ob- tained update direction for our ConFIG method is @CoFFIG = [1.5844, 0.4850, 1.4 1.4005) and the cosine similarity between 9CoFFIG and each loss-specific gradient is 7076 while IMTL-G method results in final update gradient of gIMTL-G --..7499, 0. 0.2274, -0.6566) with cosine similarity of
IMTL-G ensure GIMTL-G G (L))))
-0.7086.
A.3 EXISTENCE OF SOLUTIONS FOR CONFIG
Appendix A.2 we illustrate instances where the PCGrad and IMTL-G methods fail. This raises the question of whether ConFIG likewise exhibits fallure modes that could impede its performance.
Our ConFIG method is equivalent to solving the following system of linear equations:
(gg)). ((gg)).4()) ... (T 1m=
(28)
where the solution æ is the conflict-free gradient that should be obtained. Our ConFIG method will fail if Eq. 28 has no solution, i.e., if a conflict-free direction in the parameter space does not exist. If we use A and b to denote (L)(g)). (g)())))))))))))))))))))))))))))
.2 (g..)) )] and 1,, respectively, Eq. 28 will not have a solution if () < [[[Ab]] where R is the rank of a matrix and (A(b) is the augmented matrix of A and b. Meanwhile, since R(b)=1, and R(A) < R(AA < R + R(b), i.e.. R(A) < []] < R(A) + 1, R (A) can only be either R(A) or R(A) 1. Thus, Eq. 28 doesn't 't have solutions when R((Ab) R A) 1.
Assume that each loss-specific gradient has elements, then the shape of A and b m x h and m x (A + 1), respectively. Thus, we have:
* If A is full rank:
If m > k: R(A) =
if (A(b) is full rank: R[(A[b)) = k 1. Eq.
28 doesn't have solutions.
19

---

Published as a conference paper at ICLR 2025
if (A(b) is not full rank: [A[b]] = k. Eq. 28 has solutions.
If m I k: R(A) m. Since [[[Abb] S min m, k + 1) m. thus R[(A[])] *
R(A)+ +1)+1==0 m + 1. Eq. 28 has solutions.
If A is not full rank, i.c., R(A)
min(m, k),, we will always have p + 1 < min(m, k + 1). Thus, there is no guarantee that [[(A(b) o m + 1. Thus Eq. 28 doesn't t have solutions if R[[A|bb] p + 1, and it has solutions if R[[A[b) = p-
In practice the number of parameters in a neural network always much larger than the number of loss terms, Le.. k >> m. In this case, full rank A means R(A) m, i.c, all the loss-specific gradients are linearly independent. This is true in most situations because if the loss-specific gradi- ents were not linearly independent, we would have gj Li=1itz 04gi, where o, is the coefficient.
Since g: = /00 this would also imply L2 = E OnLi, However, the relationship between loss functions is usually not linear in practice.
To summarize, during the training procedure, we almost always have l >> m and a full rank A, en- suring the existence of conflict-fre free direction. is also worth pointing out that our ConFIG method can still provide good direction if a conflict-free direction does not exist since the Moore-Penrose pseudoinverse will offer a least squares solution for Eq. 28 making it a "conflict-free" as possible.
A.4 PROOF OF THE EQUIVALENCE
To prove the equivalence between Eq, and Eq. 4 is equivalent to prove that U/ (O(g), +
[O(ge k(((g)) 1m where KgO.
Thus, we can first calculate
[((g))
[ [O(g) 92]
+ U g2 91 )]]
((()))) [0 g2)l + (
[U(g2)
[4/ [O(g) 92)
4 [0 g2. 91
((g2) T (O(g..9)))
(29)
Then, from Eq. 19 we can obtain
M(g1)T0(gz)) g1) 1 (91.
(30)
Thus
|g [1 (2 92)
u(gi) (O((gg9)))))))))))
(31)
(0(g2.91)
From Eq. 22 we can get
10 1 (g1,9)))
(32)
Put Fq. 32 back to Eq.
31 we get
U(g:)Tu = v g2,
(33)
Similarly, we can get
U(g2) T (((,,g))))))
(34)
Put Eq. 33 and Eq. 34 back
Eq. 29
((())) (4/ (O(g), 92)] + 11 91 = (V1-V%
- S2 V1 (?(
(35)
which is
U[O +2000g..)) -
11m-
(36)
A.5 M-CONFIG AND MOMENTUM STRATEGIES
In the current study, we use the Adam optimizer to train baseline PINNs, as shown in Algorithm The green line in the algorithm can be replaced by other methods like PCGrad or the IMTL-G method. The Adam optimizer uses the first momentum m to average the recent update direction and the second momentum to rescale the learning rate of each parameter. If we want to use the momentum to replace the gradient for the ConFIG update, an intuitive approach could be to calculate the first and second momentum for each loss liss-specific gradient and use the final rescaled momentum to calculate the input for the ConFIG update, as shown in "MA-ConFIG" in Algorithm However, this approach could result in several issues:
20

---

Published as a conference paper at ICLR 2025
Algorithm ConFIG update with Adam optimizer
Require: 8o (network weights), (learning rate), S1. 92, e (Adam coefficient), <- 0 (first momentum), wo <- 0 (Second momentum),
All operations on vectors are clement-wise except G.
for + to... do
9. G L1, Vo._ L2...., Vo mm)
D. ConFIG update of gradients
m. t 8,m-
(1 B1)ge
> Update the first momentum Update the second momentum Bias corrections for the first momentum b Bias corrections for the second momentum
v: 22 (1 B2)gE
m./(1 (1- 81)
- 0t/(1 - (3)
8. 0-1 / (V
Update weights of the neural network
end for
Algorithm 4 MA-ConFIG
Require: 8g (network weights), (learning rate), 31, 32. (Adam coefficient),
Mgg.00 0 mgm.0 o] < [0, 0, 0] (first momentum).
fgg 0. 10.,,0, ... am 0] f 0, 0, 0| (Second momentum), 1 famm + [0, 1,.0,.... ,,.00,.00)
All operations on vectors are clement-wise except G.
for f +-1 1 ..... do
1%m +
l - lg: +
gi = Vo, 1 Ci
mo. + $1
fo + (1- 81)9.
Update the first momentum of g:
Va.to. + B2.
+ (1 - $2)9
> Update the second momentum of g:
Mm vigg. mggal
man m02-000
fm
Bias corrections for first momentum terms
000
1-p1
02
8- pm
(0g 0921"
8gmll
1-8 s
1-P2 f2
81 amm mgm
gum Agm
P
Bias corrections for first momentum terms
(g.92. ................................ g = /0n te'
mai
mag
D Rescale the momentum.
/o t
Pam
0: - 0,11 - (9 99.1992. ... [m)
Update weights of the neural network with ConFIG operator
end for
Inappropriate learning rate. Ideally. the Adam method will rescale each element of the up- date gradient to 1 (m ~ g. ~ g", m/ V 1). If one of the elements of the update gradient repeatedly oscillates between positive and negative, then m becomes smaller while u becomes larger, resulting in small update step (learning rate) for the corresponding pa- rameter. However, Adam's neat adaptive learning rate adjustment feature will be destroyed if we rescale the momentum before applying the ConFIG operator. This is because ConFIG will alter each parameter's learning rate again by changing the update gradient's direction and magnitude.
Numerical instability. The current study uses singular value decomposition (SVD) to calcu- late the pseudoinverse. Our experience shows that rescaling the momentum makes the input
Table I:
Test results of different momentum configurations. "MA-ConFIG" refers to the strategy with the second momentum inside the ConFIG operation, and "NaN" means the training failed due numerical stability.
M-ConFIGG
MA-ConFIG
Burgers
+ x 10
(1.549 0.362) x 10
Schrödinger
10-4
(2.625 0.087) x 10-1 NaN
[CNvLBBT]
Kovasznay
(9.777 0.347)
x 10-9
Beltrami
(7.949 + 0.384)
x 10-5
(6.658 + 0.435) x 10-3
Burgers
(1.296 + 0.013)
x 10-4
(8.511 + 7.835) x 10-3 NaN
Schrödinger
[CN,, Lo, Lr.
(1.522 = 0.581)
x 10-3
Beltrami
(9.103 = 1.831) 10-5
(5.783 0.405) X 10-3
21

---

Published as a conference paper at ICLR 2025
matrix for SVD easily ill-conditioned has too many repeated singular values, leading to training failure.
Tab. compares the performance of M-ConFIG and MA-ConFIG methods, where M-ConFIG is always stable and significantly better than the MA-ConFIG method.
Note that the M-ConFIG method should not be implemented in combination with Adam, as it is intended to replace Adam's 's calculation of momentum. Hence, our M-ConFIG implementation uses an SGD optimizer.
A.6 COMPUTATIONAL COST
To evaluate runtime performance, Fig. 15 compares the relative wall time of all methods w.r.t. the Adam baseline, highlighting the significant advantage of the M-ConFIG the training cost. The average training cost of M-ConFIG is 0.712x that of the Adam baseline for the two-term case, and 0.557x for the three-term case.
It is also worth noting that the time cost of the ConFIG method is comparable to other gradient- based methods, indicating that the pseudoinverse operation of the ConFIG method does not add significant computational overhead. This study uses the PyTorch implementation, which utilizes singular value decomposition (SVD) to calculate the pseudoinverse. Although the exact compu- tational cost depends the specific numerical algorithm, the general time complexity of SVD is (nm?² (Grasedyck) 2010) where n is the number of gradient elements (i.e., the number of neural network's s parameter) and is the number of gradients in the current study (n >> m). This is favor- able since the number of network parameters is usually much larger than the number loss terms, and the time complexity of SVD scales linearly with the former.
On the side of memory, all gradient-based based methods, including PCGrad, IMTL-G, and our ConFIG method, usually require more memory compared to the weighting method during training due the additional cost of storing loss-specific gradients for each loss term in each iteration. The complexity of this additional memory is (nm)). Meanwhile, although momentum acceleration helps in- crease the training speed, also introduces an increased memory cost with complexity of O ((m)), it also needs store the first and second momentum during the training procedure. This could be a potential challenge when dealing with large-scale problems.
Adam
ConFIGm
M-ConFIG (hrr))
IMTL-G
PCGrad
MinMax
ReLoBRaLo
1.5
Burgers
1.5
Schrödinger
1.0
1.0
0.5
0.0
0.0
(a)
Car. Caul
RN. Co. Cx)
(b)
Cv.. Dar
LNv. Co. Cxl
1.5
Kovasznay
Beltrami
1.5
1.0
1.0
0.5
0.5
0.0
[Cxx Car)
(d)
[Cx.
Cazl
[xxi Ca.
Figure 15: Relative wall time of different methods w.r.t. the Adam baseline for one training iteration.
22

---

Published as a conference paper at ICLR 2025
A.7 PDE DETAILS
ID Burgers equation. Burgers equation is a non-linear and non-trivial model equation describing shock formations. Considering spatial-temporal field u(x; L), the Burgers equation in one space dimension is
Ou 0t
Ou Ox."
+ M
(37)
dr
where y is the viscosity and set as v = 0.01/T in the current study. The t [0, 1] and x E [-1.0, 1.0] with the corresponding initial and boundary conditions of
domain is
(( 0) Sin(
(38)
((+1.0, = u(-1100,))== 0
No analytical solution available for Burgers equation with these given boundary conditions. In the current study, we use numerical solutions from PhiFlow (Holl et al. 2020) as the ground truth solution to evaluate the PINNS' performance.
ID Schrödinger equation. ID Schrödinger equation describes the evaluation of a complex field Mxx,) u(x 1) + (o( 1) in one space dimension:
Oh 10th
|4/2 = 0,
(39)
Ōt
2 dz
x E [-5.0, x/2]. It follows periodic boundary condition and a initial condition of
h(0, sech(x)),
(40)
Similarly, no analytical solution is available for the 1D Schrödinger equation with given boundary conditions. We use the numerical solution provided by deepXDE (Bajaj et al. (023) to evaluate the performance of trained PINNs.
2D Kovasznay flow and 3D Beltrami flow. Kovasznay and Beltrami flow are special solutions for Navier-Stokes equations which describe the fluid flow:
du Ot + (u V)u = VPp
1
V
Re
(41)
v
where is the velocity vector, p is the pressure and Re is the Reynolds number. Kovasznay flow is a steady case in two space dimensions where the transient term 0u/Ot equals 0. The analytical solution of the velocity uxx,y) v) [uz (I,, w), uz y)] and pressure p(x,y) y) is (Kovasznay 1948
Truesdell 1954)
(1 y) = - ed cos cos(()y))
My (x, v)
2z
a ee sin(2Tv))
(42)
p(x,v) v)
1 2xx)
where 1
+ 4T2.
Re :. x E [-0.5, 1] and y [-0.5, Meanwhile,
2y V 1
Beltrami flow is an unsteady case in three space dimensions where the velocity u(r, 2, 2, = [1, (2, y, U,2.6. f). u. (z. y, 2, t), uz (z. y, 2, t)] and pressure p(x, 2, follows the analytical solution
of (Gromeka 1881)
2,t)
[ear sin(ay + dz) + -= cos + dy)] dt -a Jeav sin(az + dr) + e0 cos + dz) e -dit -a [eaz sin(ar + dy) + e cos(az + dx) -
(,,
(z(2,,,2,t)
p(r,8,2,()=
- --aa [620z + 20y + 2ax
(43)
+2 sin(ax + dy) cos az + dr) dr)e
+2sin + dz) cos(ax ar + d
+22 sin + dx) cos(ay + dz (x++()))))))) 26ft
where a = d = 1 when Re = in our configuration. The simulation domain size is x E [-1,1]] y E 1], 2 E [1,, and t [0, 1]. The boundary conditions of Kovasznay and Beltrami flow are Dirichlet type. which follows the values of analytical solutions. For Beltrami flow, the initial condition also follows the values of the analytical solutions.
23

---

Published as a conference paper at ICLR 2025
A.8 PINN-BASED EXAMPLE SOLUTIONS
This section provides examples of the solution domain for each PINN case. Figures 16 18
23
22d ]2] show the mean predictions of the networks compared to the ground truth, the standard devia- tion, and the squared error distribution for each PINN case. Additionally, Figures] 17 19 and 21 offer detailed comparison of sample line in the solution domain for the Burgers equation, Schrödinger equation, and Kovasznay flow case.
0.5
0.0
0.5
-0.5
0.0
0.5
Ground truth
CanFIG
0.005
0,010
0.015
0.00
0.01
0.02
24
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
SE
Figure 16: The solution domain of Burgers Equation with corresponding standard deviation(a) and square error(SE) of the ConFIG method's prediction.
t 0.25
0.50
0.75
2
Ground truth ConFIG
Ground truth
Ground truth ConFIG
H
ConFIG
()
0
45
Figure 17: The distribution of in the solution domain of Burgers equation.
24

---

Published as a conference paper at ICLR 2025
3.93
1:38 0.69 2.75 0.01
ConFIG Ground Truth
0.00 0.02 0.04 0.06 0.00 0.02 0.05 0.07 0.00 0.01 0.01 0.02
0
0.00
0.02 0.05 0.07 0.00 0.03 0.06 0.08 0.00 0.00 0.00 01
0
1x
Figure 18: The solution domain of Schrödinger Equation with corresponding standard deviation( and square error(SE) of the ConFIG method's prediction.
0.79
Ground truth ConFIG
Ground truth ConFIG
Ground truth ConFIG
2
2
0-
0-
-5
-5
-5
0
5
Figure 19; The distribution of [h] in the solution domain of Schrödinger equation.
0.618
1.001
2.620
0.248 0.000 0 249
0.811 0.192 0.427
ConF G Ground Truth
G
3.5e-06
1.1e-03 3.4e-07
4.4e-04 1.5e-05
1.
3.0e-16
8 8:3e-07 2 4e-18
3.1e-07
9.5e-15
2.6e-07
w
Figure 20: The solution domain of Kovasznay Flow with corresponding standard deviation(g) and square error(SE) of the ConFIG method's prediction.
25

---

Published as a conference paper at ICLR 2025
0.25
0.00
0.50
Ground truth
Ground truth ConFIG
Ground truth ConFiG
ConFIG
3-
2
Figure 21: The distribution of ug in the solution domain of Kovasznay Flow.
.2.32
0.441
1.449
-2.332
-0.443
1.446
Ground truth
ConFIG
Ux
4.90e-05
7.60a-03
1.52e-02
-9.75e-03
-7.84e-04
8.18e-03
Oue
SE.,
Figure 22: The solution domain of Beltrami Flow when 0.5 with corresponding standard deviation(7)) and square error(SE) of the ConFIG method's prediction.
26

---

Published as a conference paper at ICLR 2025
A.9 PINN EVALUATIONS
The exact numerical values and corresponding standard deviations of the PINN experiments are presented in this section. Tables 2 through 5 display the values obtained with the same number of training epochs as depicted in Fig. and Fig. 6 Additionally, Tables 6 through 9 showcase the values obtained with identical wall time, corresponding to the data presented in Fig.
Table 2: Test MSE of PINNs trained for Burgers equation. All values are scaled with 44
Lv, LBI]
[CN; Ls, Li]
Adam
1.484 + 0.061
PCGrad
1.344 = 0.019
1.279 = 0.008
IMTL-G
1.339 + 0.024
22.478 = 15.008
MinMax
1.889 + 0.143
1.582 = 0.051
ReLoBRaLo
1,419 + 0.053
1.402 = 0.034
LRA
353.796 + 114.972
444.603 + 16.695
ConFIG
1.308 + 0.008
1.291 + 0.039
M-ConFIG
1.277 + 0.035
1.296 = 0.013
Table 3: Test MSE of PINNs trained for Schrödinger equation. All values scaled with 10-1
[CN, LOz.
[CN: L8, Cx.]
Adam
3.383
1.178
PCGrad
1.621 + 0.547
0.931 + 0.028
IMTL-G
7.891 3.008
2504.282
588.500
MinMax
45.255 + 1.535
45.068 + 5.292
ReLoBRaLo
3.603 2.165
2.756 + 0.098
ConFIG
0.643 0.227
1.455 + 0.455
M-ConFIG
4.292 + 1.863
15.217 + 5.807
Table Test MSE of PINNs trained for Kovasznay flow. All values are scaled with 10-7
[Lv, Lg]
Adam
1.044 + 0.405
PCGrad
0.799 = 0.083
IMTL-G
0.096 0.012
MinMax
7.743 + 1.197
ReLoBRaLoo
1.148 + 0.359
LRA
3.595 + 3.582
ConFIG
0.126 0.048
M-ConFIG
0.098 + 0.003
27

---

Published as a conference paper at ICLR 2025
Table 5: Test MSE of PINNs trained for Beltrami flow. All values are scaled with 10 -
[CN, LBT]
[Lv; LT,
Adam
1.112 + 0.119
PCGrad
1.037
0.119
0.757 + 0.082
IMTL-G
0.568 + 0.078
0.932 + 0.228
MinMax
0.100
ReLoBRaLo
1.197 0.029
1.028 = 0.107
LRA
0.708 0.047
0.843 = 0.146
ConFIG
0.617 + 0.112
0.571 + 0.090
M-ConFIG
0.795 + 0.038
0.910 + 0.183
Table 6: Test MSE of PINNs trained for Burgers equation. All values are collected after the same wall time and scaled with 10-4
[Cx, LBr]
LB,
Adam
1,513 + 0.079
1.546 = 0.100
PCGrad
+.0022
1.300 = 0.023
IMTL-G
1.356 + 0.046
82.446 = 76.016
MinMax
3.058 0.379
2.030 = 0.153
ReLoBRaLo
1.449 + 0.033
1.492 = 0.099
LRA
376.449 + 100.599
470.108 = 28.172
ConFIG
1.330 + 0.015
1.665
0.186
M-ConFIG
1.277 + 0.035
1.296 = 0.013
Table 7: Test MSE of PINNs trained for Schrüdinger equation. All values are collected after the same wall time and scaled with 10-3
[Cv, Laz]
[Cx, La, Lz]
Adam
2.169 + 0.584
5.832 1.421
PCGrad
1.505 - 0.559
2,271 0.205
IMTL-G
8.844 + 4.263
291.037 3.682
MinMax
28.195 1.616
49.635 2.888
ReLoBRaLo
2.369 0.829
15.230 7.691
LRA
316.549 + 8.696
315.606 = 8.969
ConFIG
0.586 + 0.256
4.636 + 0.932
M-ConFIG
0.429 + 0.186
1.522 0.581
Table Test MSE of PINNs trained for Kovasznay flow. All values are scaled with 10-7
[Lv, Lz)
Adam
4.037 +: 1.078
PCGrad
2.723 + 0.421
IMTL-G
0.419 + 0.058
MinMax
261.628 + 43.303
ReLoBRaLo
3.291 + 0.847
LRA
13.554 + 13.773
ConFIG
0.631 + 0.194
M-ConFIG
0.098 + 0.003
28

---

Published as a conference paper at ICLR 2025
Table 9: Test MSE of PINNs trained for Beltrami flow. All values collected after the same wall time and scaled with 10-4
[AN, LBr)
[Lv: LB,
Adam
2.318 + 0.270
3.648 + 0.244
PCGrad
2.206 0.180
2.351 + 0.212
IMTL-G
1.054
2.194
MinMax
5.422 0.300
15.754
ReLoBRaLo
2.605 + 0.056
2.769 + 0.202
LRA
1.194 0.139
2.259 + 0.812
ConFIG
1.130 + 0.138
1.451 + 0.181 0.910 + 0.183
M-ConFIG
0.795 + 0.038
Table 10: Test MSE of PINNs trained for Burgers equation with the ConFIG method using different direction weights. All values are scaled with 10-4
[Cx, CBz]
Cz,
MinMax
2.444 + 0.312 1.310 0.007
45.068 = 5.292
ReLoBRaLo
1.315 0.048
LRA
449.658 = 42.392
657.258 = 45.487
Equal
1.308 + 0.008
1.291 + 0.039
Table 11: Test MSE of PINNs trained for Schrodinger equation with the ConFIG method using different direction weights. All values are scaled with 10
[Cx, LBr]
[Ax: LB, Cr]
MinMax
2.444 + 0.312
3.733 + 1.351
ReLoBRaLo
1.048 + 0,404
2.829 + 1.444
LRA
25.417 6.766
3118.378 + 45.723
Equal
6.429 + 2.269
1.455 + 0.455
Table 12: Test MSE of PINNs trained for Kovasznay flow with the ConFIG method using different direction weights. All values are scaled with 10-7
[CN, Loz
MinMax
7.487 + 2.545
ReLoBRaLo
0.104 + 0.008
LRA
(9.047 + 2.052) x 1
Equal
0.126 + 0.048
Table 13: Test MSE PINNs trained for Beltrami flow with the ConFIG method using different direction weights. All values are scaled with 10-4
[Cv, Laz]
MinMax
1.903 + 0.271
1.597 + 0.039
ReLoBRaLo
0.658 + 0.157
0.575 + 0.059
LRA
155.617 + 55.853
15.754 + 5.049
Equal
0.617 + 0.112
0.571
0.090
29

---

Published as a conference paper at ICLR 2025
A.10 RESULTS OF CELEBA MTL EXPERIMENTS
Table 14 19 presents the numerical values and corresponding standard deviations for the CelebA MTL experiments, aligning with the results in Fig. 1 and Fig. [2] The best performance is deter- mined by selecting the best average performance across different tasks during training, which is also the value shown in Fig. [I The average performance calculated based on the results from the last 5 epochs, during which most methods have converged. For both best and average performance, our ConFIG method surpasses all other methods in M R metrics. For the performance in F1, it tied for first with the FAMO method in best performance and ranked second in the average performance.
Table 14: M performance of different methods in CelebA MTL experiments
Best performance
Average performance
PCGrad
6.425 + 0.595
5.475
IMTL-G
6.058 + 0.671
5
FAMO
6.233 = 0.085
4
NASHMTL
6.158 = 0.242
5.612 1.237
RLW
6.300 + 0.602
CAGrad
6.767 + 0.274
7.387 + 1.062
GRADDROP
6.733 0.309
+
DWA
6,908 + 0.450
8.500 0.000
LS
7.383 = 0.112
8.675 + 0.075
UW
7.692 + 0.051
8.238 + 0.438
M-ConFIG 30 ConFIG
5.833 + 0.342
7.175 + 0.525
5.508 + 0.392
4.362 + 0.812
Table 15: Fi performance of different methods in CelebA MTL experiments
Best performance
Average performance
PCGrad
0.681 + 0.003
0.665 + 0.007
IMTL-G
0.680 + 0.003
0.660 + 0.003
FAMO
0.686 = 0.004
0.673 + 0.002
NASHMTL
0.683 = 0.006
0.642 + 0.026
RLW
0.680 = 0.002
0.663 = 0.006
CAGrad
0.672 + 0.010
0.649 + 0.008
GRADDROP
0.661 + 0.005
0.644 0.010
DWA
0.672 + 0.003
1.04000000.
LS
0.676 + 0.004
0.639 + 0.004
UW
0.675 = 0.009
= 0.641
M-ConFIG 30
0.694 + 0.005
0.655 + 0.006
ConFIG
0.686 + 0.006
0.671 + 0.003
Table 16: The F, performance of the M-ConFIG and ConFIG method with different numbers of tasks in the CelebA experiment.
Best performance
Average performance
uupdates
ConFIG
M-ConFIG
ConFIG
M-ConFIG
0.697 + 0.007
0.664 + 0.005
0.645 + 0.006
0.638 0.003
10
0.695 + 0.006
0.536 = 0.057
0.668 = 0.013
0.488 = 0.055
20
0.684 + 0.008
0.455 + 0.032
0.660 - 0.007
0.408
0.015
30
0.667
+ 0.006
0.429 + 0.023
0.648 + 0.006
0.383 + 0.025
40
0.686 + 0.006
0.423 0.026
0.671 = 0.003
0.383 0.037
30

---

Published as a conference paper at ICLR 2025
Table 17: The training time of the M-ConFIG and ConFIG method with different numbers of tasks the CelebA experiment.
Thusks
ConFIG
M-ConFIG
5
180.333 0.471
67.436 = 0,267
10
288.000 + 0.816
73.588 = 1.262
20
561 1610000 0.816
84.252 + 0.919
30
861.00 151.000 + 0.816
110.454 + 0.487
40
1179.000 + 2.449
167.423 + 0.427
Table 18: The Fi performance METof the M-ConFIG and ConFIG method with different numbers of gradient updates in the CelebA experiment (40 tasks).
Best performance
Average performance
Tuuddttes
ConFIG
M-ConFIG
ConFIG
M-ConFIG
0.423 0.026
0.383 0.037
0.458 + 0.006
0.406 = 0.008
0.570 + 0.049
0.517 0.049
0.686 + 0.006
0.671 + 0.003
20
0.681 = 0.006
0.668 0.008
0.694 = 0.005
0.678 0.006
40
0.682 + 0.007
0.654 0.015
Table 19: The training time of the M-ConFIG and ConFIG method with different numbers of gradi- ent updates in the CelebA experiment (40 tasks).
Rupdates
ConFIG
M-ConFIG
310.000 + 0.000
492.667 + 1.247
10
723.000 0.816
2090.618 + 5.627
20
1185.667 = 6.018
30
1647.667 = 7.587
40
2084.667 + 24.635
31

---

Published as a conference paper at ICLR 2025
A.11 ADDITIONAL RESULTS ON CHALLENGING PDES
To evaluate the performance of our method for challenging and high-dimensiones test cases, we ap- ply it to three problems from recent benchmark for PINNs (Hao al. [002). In the following, we perform comprehensive investigation of the performance of our method in solving these problems.
A.11.1 HIGH DIMENSIONAL PDES
We introduce the N-dimensional Poisson equation as the test case for high- digendimensional problems. The governing PDE is
2
u
sin
2 mi)
(44)
where n is the number of dimensionality. In the current experiment, we choose n = and set the spatial domain as [0, 11$ following the configuration of Hao al. (2023). The ground truth solution is
sin n.) 2
(45)
The PNd problem employs Dirichlet boundary conditions with the boundary value equal to the analytical solution.
Tab. 20 and Fig. [23] summarize the test results of different methods. Our M-ConFIG method ranks first among methods, followed by the ConFIG methods, showing the superiority of our methods in dealing with high-dimensional problems.
Table 20: Test MSE of PINNs trained for PNd equation. All values x x10
[Cv,
Adam
1.916 + 0.284
PCGrad
1.313 + 0.097
IMTL-G
0.520 0.123
MinMax
4.604 +
ReLoBRaLo
2.265 = 0.263
LRA
0.639 0.340
ConFIG
0.461 + 0.141
M-ConFIG
0.415 + 0.052
1.1
Ground truth
3.5
3.5g
3 3
2.8
280
2
2.1
21
Figure 23: The solution domain of PNd problem [x. y, 0.5, 0.5, 0.5] plane.
A.11.2 MULTI-SCALE PROBLEMS
We choose the multi-scale heat transfer (HeatMS) problem as a strongly anisotropic test case. We set different heat-transfer coefficients in different spatial directions to give the solution different scales in each direction. Following the configuration of Hao et al. (2023), the governing equation
du
pu
1 8
= 0,
(46)
01
(500T) )2
=2 0y
32

---

Published as a conference paper at ICLR 2025
with the initial condition of u(2,, y. 0) = sin (2002))))))))) sin(Ty) and boundary condition of u(z, y. 0. The spatial time domain is x E [0, 1], y E [0, 1] and E [0,5) 5). Fig. 24 illustrats a sample solution of the ground truth when t 3.0s.
Tab. and Fig. 25 show the predictions by different methods. While all of the methods struggle to fully resolve the anisotropic solutions, shown in Fig. 25 M-ConFIG method still ranks first. This test case illustrates that the scaling issues of anisotropic PDEs can not be solved purely by finding better balance between different loss terms. Nonetheless, ConFIG fares on-par with other methods, and the momentum terms of M-ConFIG help to partially address the scaling of the HeatMS test case.
Table 21: Test MSE of PINNs trained for HeatMS problem. All values scaled with 10-3
[CN, LbT]
[LN, CT.
Adam
7.612 + 0.004
PCGrad
7.600 + 0.012
7.603 + 0.012
IMTL-G
6.693 + 0.241
38.529
+ 43.666
MinMax
7.595 = 0.055
7.243 0.458
ReLoBRaLo
7.585 = 0.039
7.558 = 0.041
LRA
7.654 + 0.004
7.652 + 0.000
ConFIG
7.529 + 0.074
7.585 = 0.020
M-ConFIG
5.978 + 0.092
7.147 + 0.001
0.01
0.02
0.00
-0.92
Figure 24: The ground truth solution of HeatMS problem when = 3.0s.
0.3
0.4
0.2
1.1
0011
0.1
-.3
0.2
0.1
1.8
0.7
0.1
0.1
0.5
0
RoLoBRalo
0.2
01
0.0
08
0.2
.02
001
12
0.5
Figure 25: The predictions of PINNs for HeatMS problem when t = 3.0s (two-loss scenario, values scaled with 10-2).
1.13.3 CHAOTIC PROBLEMS
To test the potential limitations the proposed methods, we target the KS equation as a representa- tive of chaotic problems. Starting from an analytic initial state, the solution of the KS equation will gradually develop into a chaotic state due to the strong nonlinearity, dissipative, and destabilizing effects. Following the configuration of Hao et al. (2023). the governing equation is
33

---

Published as a conference paper at ICLR 2025
Du
100 du
100 8.u
100 0'u
16 Or -0,
(47)
16 dr
16 Ox
with initial condition of u(a, 0) cos(2)(1 sin(x)) and periodic boundary conditions. The spatial time domain is æ E [0, 2m] and L E [0, l].
Table 22: Test MSE of PINNs for the KS equation.
[AN: LAr]
[Cv, LB, Cr]
Adam
1.094 + 0.002
1.094 = 0.002
PCGrad
1.089 0.008
1.061 + 0.002
IMTL-G
1.110 + 0.016
1.089 + 0.031
MinMax
1.072 0.000
1.084 0.007
ReLoBRaLo
1.074 0.019
1.080 = 0.016
LRA
1.099 + 0.039
1.098 = 0.027
ConFIG
1.089 + 0.004
1.062 + 0.007
M-ConFIG
1.123 + 0.011
1.118 = 0.008
Tab. [2 shows the predictions by different methods. The evaluation shows that no methods succeeds capture the transition of chaos and generate smooth solutions. This agrees with the conclusions in Hao et al. (2023)'s, ss where likewise no method under consideration manages to converge to an accurate solution.
The inherent challenge of solving the KS equations is combination of several factors. On the one hand, the negative diffusion term Ouu/Oz² injects energy into the system, inducing instability and complex structures. On the other hand, The dissipative term 0'u/Oz" removes energy at small scales, formulating energy cascade across different scales. This energy cascade causes huge diffi culties for numerical methods since the fine-scale structures must be resolved accurately. From the PINN side, this requires the network to have the ability to estimate the derivatives accurately on different scales via auto differentiation. The balance between different loss terms, all the bench- mark methods are trying to achieve, is not helpful in addressing this fundamental issue. Hence, this test case serves as a failure case, highlighting that improved optimizers alone do not suffice to address these challenges. It will be an interesting topic of future work to evaluate their effectiveness in combination with other changes, such improvements the architecture side.
A.12 TRAINING DETAILS
The training of PINNs in the current research follows the established conventions. The neural networks are fully connected with 4 hidden layers and 50 channels per layer. The activation function is the tanh function, and all weights are initialized with Xavier initialization (Glorot & Bengio 2010). Data points are sampled using hyperrhee sampling and updated in each iter- ation. The extended training run case shown in Fig. 10 uses constant learning rate of 10-4, All other cases follow cosine decay strategy with the initial and final learning rate of 10 33 and 10 respectively. We also add learning rate warm-up of 100 epochs for each training. All the methods except M-ConFIG use the Adam optimizer. The hyper-parameters of the Adam optimizer are set as 31=0.9, 82 0.999, and 10-8 respectively. The number of data points and training epochs for each case are listed in Tab. 23
PINNs.
MTL. Our experiments are based on the official test code of the FAMO method, and our ConFIG method implemented in the corresponding framework. For the details of the configurations, please refer the original FAMO paper (Liu et al. (202)) and its official repository: hhtpps: //g/ttubttbb com/Crania XIX/FAMM (MIT License..
Compute resources. All the experiments this study were conducted using an NVIDIA RTX A5000 GPU with 24 GB of memory. Each PINN experiment completes training within a few hours on a typical GPU with more than 4 GB of memory. For the CelebA MTL test, a GPU with more than 12 GB of memory is required, and single training takes 1-2 days.
34

---

Published as a conference paper at ICLR 2025
Table 23: The number of data points and training epochs for PINNs' experiments.
Number of data points
Epochs
ng
n2
Burgers equation
10000
250
250
3 x 104
Schrödinger equation
20000
500
500
10a
Kovasznay flow
20000
1000
10a
Beltrami flow
25000
5000
5000
10s
PNd
20000
5000
10a
HeatMS
20000
2000
2000
10a
KS
20000
500
500
10
A.13 ABLATION STUDY ON TRAINING HYPERPARAMETER
Although utilize momentum based optimizer for the training, the learning rate may still affect the training process, especially considering that our ConFIG and M-ConFIG methods consistently change the gradient vector during the training. Thus, perform ablation study different learning rates. As shown in Tab. 24d Tab. 25 our methods consistently outperform other methods with different learning rate configurations.
Table 24: The performance of different methods with different cosine decay learning rates in Burgers two-loss test
10-3 - 10-4
10-1 - 10-5
Adam
1.484
3.076 = 1.201
PCGrad
1.344
2.223 = 0.355
IMTL-G
1.339 + 0.024
1.688 = 0.018
MinMax
1.889 0.143
3.980 = 0.798
ReLoBRaLo
+ +
5.057 = 1.821
LRA
353.796 + 114.972
819.730 + 56.925
ConFIG
1.308 + 0.008
1.887 = 0.283
M-ConFIG
1.277 + 0.035
1.681 + 0.106
Table 25: The performance of different methods with different constant learning rates in Burgers
two-loss test
7 = 10 33
y= 10
7 = 10 5
Adam
1.398 0.021
3.076 + 1.201
420.511 + 95.578
PCGrad
1.398 0.018
2.223 + 0.355
51.664
IMTL-G
1.385 + 0.042
1.688 + 0.018
192.151
123.903
MinMax
1.889 0.120
3.980 0.798
= 7.078+1.677
ReLoBRaLo
1.477 + 0.061
5.057 = 1.821
10.333 1.905
LRA
367.944 = 98.322
819.730 56.925
895.965 1.671
ConFIG
1.373 0.006
1.887 = 0.283
103.239 + 40.287
M-ConFIG
1.354 + 0.019
1.681 = 0.106
4.097 = 1.310
In the current study, the points are sampled dynamically from the internal domain and boundaries during training. Thus, the number of data samples represents the training batch size. Meanwhile, each loss-specific gradient evaluated through the data samples at the internal domain and bound- aries. Thus, the number of sample points and the relative ratio between the number of points the boundary and the internal domain may affect the quality of the gradient, further affecting the final training results. Here, also perform ablation study on the data samples, and the results
35

---

Published as a conference paper at ICLR 2025
summarized in Fig. 26 Tab. 26 and Tab. 27 Our methods consistently outperform other methods
with different configurations of data samples, showing the robustness of our methods.
20
ConFIG
15
M-ConFIGagee
RI(%)
PCGrad
IMTL-G
MinMax
ReLoBRaLo
-10
-10%
LRA
10000 fy. 250
5000
VA 20000 My 500
N. 5000 Ni flg 250
Wly 200000
my 125
Figure 26: The relative improvements of different methods with different numbers and different ratios of data samples during training
Table 26: The performance of different methods with different numbers of data samples during training
nN 5000
n. 10000
nW 20000 fg = nr = 500
ng = ni = 125
ng = fz = 250
Adam
1.494 + 0.060
1.484 + 0.061
1.438 + 0.031
PCGrad
1.350 0.015
1.344 0.019
1.331 0.037
IMTL-G
1.368 + 0.012
1.339 + 0.024
1.275 + 0.003
MinMax
1.943 0.212
1.889 0.143
1.846 0.135
ReLoBRaLo
1.478 + 0.067
1.419 + 0.053
1.381 + 0.043
LRA
340.161 135.263
353.796
114.972
374.574 = 84.340
ConFIG
1.307 0.014
1.308 0.008
1.295 0.014
M-ConFIG
1.291 0.017
1.277 + 0.035
1.262 0.004
Table 27: The performance of different methods with different ratios of data samples during training
nw 5000
NN 10000 ng = ng = 250
n. 20000 ng = ni = 250
ng = ny = 250
Adam
1.470 + 0.012
1.484 0.061
1.455 + 0.044
PCGrad
1.358 + 0.026
1.344 = 0.019
1.355 0.013
IMTL-G
1.328 + 0.036
1.339 + 0.024
1.342 + 0.027
MinMax
1.963 + 0.133
1.889 = 0.143
1.843 0.152
ReLoBRaLo
1.469 = 0.014
1.419 + 0.053
1.393 = 0.052
LRA
362.388 + 110.455
353.796 + 114.972
350.266 + 127.639
ConFIG
1.298 + 0.014
1.308 + 0.008
1.300 + 0.010
M-ConFIG
1.289 0.015
1.277 0.035
1.265 0.026
36


Reviews:
[{"review_id": "3Yxlxu8Cd1", "reviewer": "Reviewer_5ieJ", "summary": "Motivated by the discrepancy between the residual and data loss terms in Physics-Informed Neural Networks (PINNs), the authors propose a \"conflict-free\" gradient update for multi-task learning, such that the resulting optimization direction minimizes all individual loss terms, avoiding local minima induced by one loss term. Such optimization step is calculated by ensuring positive cosine similarity between the update direction and the individual gradients of each loss term and (in most cases) a uniform decay rate for all losses. The authors also propose a momentum-based algorithm to alleviate the computational costs.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.  The experiments thoroughly evaluate the proposed method from various aspects and for different problems.\n    - The effectiveness (in terms of accuracy) of both ConFIG and Momentum(M)-ConFIG algorithms is demonstrated compared to other gradient adjustment methods, loss weighting methods for PINNs, and Adam.\n    - While M-ConFIG falls short of the original ConFIG, its performance in computation-constrained scenarios is well shown.\n2. The motivating problem from PINNs is a significant obstacle to wider adoption of them, and the proposed algorithm paves the way for their application to more problems in Physics.\n3. The paper is well-written and accompanied by a reasonable amount of experiments, maths, and pseudo-code.", "weaknesses": "1. Although the experiments are already extensive, the paper can benefit from experiments with other PDEs (e.g. KS, Turbulence) and boundary conditions (e.g. Neumann) from the studied loss weighting baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OCDurspI3S", "reviewer": "Reviewer_Gt6s", "summary": "This paper proposes a novel method (ConFIG) to deal with multi-task problems, and specifically focus on the task of training physics-informed neural networks (PINNs). The main motivation behind the method is that multiple loss functions in their formulation (PDE residuals, boundary / initial conditions) often conflict and lead to suboptimal optimizations. To overcome this limitations, ConFIG ensures that the update direction is conflict free by maintaining a positive-dot product between the final direction and the individual loss-specific gradients. Additionally, the paper introduces a momentum-based version (M-ConFIG) which reduces computational efficiency while maintaining training efficiency. The proposed method is contrasted against other loss aggregation schemes on several PDE problems and a multi-task learning benchmark, and demonstrates improved performance over existing methods.", "questions": "In addition to the points raised in weaknesses, it would be good if the authors can address the following:\n- It seems like this approach has been primarily studied in the context of forward problems. Have you performed any studies on inverse problems as well?\n- Additionally, I'm curious how the more recent pre-print of Jacobian Descent for Multi-Objective Optimization (Quinton & Rey) ties in, and compares against your work. Again, they compare against several MTL aggregators and propose a new strategy that claims to avoid conflicts with any of the individual gradients.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper addresses the important issue of conflicting directions in gradient updates in the training of PINNs. The proposed approach is novel and seems to combine the best of other MTL methods that have been applied in the context of PINNs such as PCGrad and IMTL-G. The proposed formulation seems well-formulated and is theoretically sound. Further, the introduction of M-ConFIG as a method to accelerate training while avoiding the evaluation of all the individual gradients at every step is a great practical contribution. Additionally, the appendices on this aspect are also thorough and discuss the reason as to why using Adam with ConFIG wouldn't work.\n\n- The experiments also reveal a better performance than other baselines across different PDE benchmarks. Moreover, the method also performs well when applied to the standard MTL CelebA benchmark revealing broader applicability. The experiments also showed better convergence when compared against computational time of other optimization schemes further improving the applicability.", "weaknesses": "- The experiments that are peformed in the paper compare against other MTL aggregation schemes for the proposed benchmark problems. However, there are several challenging benchmark problems for PINNs that have been established in other works (such as PirateNets (Wang et al.), PINNacle (Hao et al.),  Expert's Guide to PINNs (Wang et al.)). Although, they do use other enhancements such as an improved network architecture / improved encodings / imposing causality, it would be interesting to see if these enhancements would further improve on the current state of the art results.\n\n- The hyperparameter tuning and ablation studies in the paper could be more extensive. For instance, the authors claim that the learning rate was fixed to $10^{-4}$ for the long-training or the use of a $10^{-3} \\to 10^{-4}$ decay. But they don't seem to study the influence of this chosen rate on the final relative L2 error that has been obtained. It would strengthen the position of the paper to study this aspect as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7Q2bw9l1i", "reviewer": "Reviewer_PVNV", "summary": "This paper addresses the challenge of gradient conflicts in Physics-Informed Neural Networks (PINNs) and the tendency of optimization to get stuck in the local minima of specific loss terms. The authors highlight that the gradient magnitude of PDE residuals typically dominates that of initial and boundary condition losses, leading to biased final updates. Their proposed ConFIG method leverages a momentum-based approach that reduces computational costs by alternating the backpropagation of loss-specific gradients, ensuring conflict-free updates and improved convergence.", "questions": "In addition to what I mentioned in weaknesses, I would like to know how this type of models can be compared to traditional methods to improve PINN training typically involves adjusting the weights for PDE residuals and loss terms for initial and boundary conditions in terms of convergence and performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The optimization of different loss functions in PINNs is critical, as it can influence whether PINNs succeed or fail in solving forward problems. The exploration of Multi-Task Learning (MTL) strategies for improving PINNs training is innovative and may inspire future methodologies.\n- The problem setup and method are clearly explained, making the paper accessible and informative. The theoretical background is well-supported by proofs and discussions.\n- The experimental design is good, but it needs some adjustments.\n- Given the difficulty of training PINNs due to conflicting gradient directions and numerical issues, the ConFIG method can be a good direction for solving these problems.", "weaknesses": "- The examples explored in this study are relatively simple and can be solved with standard methods like Adam. It would be insightful to evaluate the proposed approach on more complex, chaotic PDEs, such as the Kuramoto-Sivashinsky equation, which poses significant challenges due to its non-linear and chaotic behavior.\n- The study omits comparisons with certain MTL strategies, such as FAMO, which could provide further context to the method’s effectiveness in PINNs. Additionally, using only the CelebA dataset to demonstrate performance in MTL may not be comprehensive enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6CSD9p2jHw", "reviewer": "Reviewer_4Pd7", "summary": "The paper introduces a novel method, ConFIG (Conflict-Free Inverse Gradients), aimed at addressing gradient conflicts that arise during the training of Physics-Informed Neural Networks (PINNs). The primary challenge in training PINNs lies in the conflicting gradient directions from multiple loss terms, particularly those associated with initial/boundary conditions and the physics equations, which can impede learning and optimization.  The paper proposes the ConFIG method, which ensures that the final update gradient has a positive dot product with each loss-specific gradient, eliminating conflicts and enabling consistent optimization across all loss terms. The method adjusts gradient magnitudes dynamically based on conflict levels and maintains uniform optimization rates for all loss terms. The method provides a robust and conflict-free optimization framework for PINNs, and its application extends to multi-task learning, showcasing improved performance over existing methods.", "questions": "1. What is the computational complexity of the pseudoinverse operation in ConFIG, and how does it scale with larger problem sizes?\n\n2. It would be better if the authors could test ConFIG on high-dimensional or multi-scale PDE problems, as seen in benchmarks like PINNacle? If so, how does it perform?\n\n3. The authors are expected to provide more insight into how sensitive ConFIG is to hyperparameters like learning rate and batch size? \n\n4. The authors are expected to evaluat ConFIG’s scalability and efficiency for large-scale industrial applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes a notable contribution to the field of PINN training by addressing a fundamental challenge with an original and technically sound approach.\n\nOriginality: The ConFIG method introduces a novel solution to gradient conflict resolution in PINNs. It effectively adapts ideas from multi-task learning (MTL) and continual learning (CL) to address gradient conflicts, which is relatively unexplored in the PINN domain.\n\nQuality: The paper demonstrates strong technical quality with rigorous mathematical proofs and comprehensive experimental validation across multiple benchmark problems. While the method’s effectiveness is well-supported, additional comparisons with state-of-the-art methods and scalability analysis would further bolster its quality.\n\nClarity: The paper is generally well-organized and clear, though certain sections, particularly those with heavy mathematical notation, could be simplified for broader accessibility. \n\nSignificance: The method addresses a critical challenge in PINN training, with broad implications for fields like fluid dynamics, electromagnetics, and beyond.", "weaknesses": "The paper would significantly increase its rigor and relevance by including more comprehensive comparisons, and the detailed comments are \n1. Limited Benchmarking: The evaluation is narrow, focusing on standard PDEs like Burgers equations, without addressing more complex, real-world PDE challenges (e.g., multi-scale phenomena, high-dimensionality). It would be helpful if the authors could broaden the evaluation to include more diverse and complex PDEs, as seen in benchmarks like PINNacle (Hao et al., 2024), to demonstrate the robustness of ConFIG across a wider range of problems.\n\n2. Comparison with Advanced Methods: The paper primarily compares ConFIG to basic methods (e.g., PCGrad, IMTL-G), without considering recent, more sophisticated approaches in PINN optimization. The authors are expect to include comparisons with advanced methods.\n\n3. Scalability and Efficiency: The scalability of ConFIG, especially regarding the computational cost is not fully explored. It would be better if the authors could provide a detailed analysis of runtime and memory efficiency, particularly for large-scale or high-dimensional problems, to validate ConFIG’s scalability.\n\n4. Ablation and Sensitivity Studies: There’s a lack of ablation studies to assess the impact of key hyperparameters like learning rate and gradient scaling. The authors could perform more ablation studies to better understand the effect of different components and configurations on ConFIG’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)