ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks

## Abstract

Abstract The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. The proposed method is evaluated across a range of challenging PINN scenarios, consistently showing superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG .

## 1 Introduction

Efficiently solving partial differential equations (PDEs) is crucial for various scientific fields such as fluid dynamics, electromagnetics, and financial mathematics. However, the nonlinear and high-dimensional PDEs often present significant challenges to the stability and convergence of traditional numerical methods.
With the advent of deep learning, there is a strongly growing interest in using this technology to solve
PDEs [, ].
In this context, Physics Informed Neural Networks (PINNs) [, ]
leverage networks as a continuous and differentiable ansatz for the underlying physics.

PINNs use auto-differentiation of coordinate-based neural networks, i.e., implicit neural representation (INR), to approximate PDE derivatives. The residuals of the PDEs, along with boundary and initial conditions, are treated as loss terms and penalized during training to achieve a physically plausible solution. Despite their widespread use, training PINNs is a well-recognized challenge [, , , , ] due to several possible factors like unbalanced back-propagated gradients from numerical stiffness [], different convergence rates among loss terms [], PDE-based soft constraints [], poor initialization [], and suboptimal sampling strategies []. Traditional methods to improve PINN training typically involve adjusting the weights for PDE residuals and loss terms for initial/boundary conditions [, , ]. However, although these methods claim to have better solution accuracy, there is currently no consensus on the optimal weighting strategy in PINN training.

Meanwhile, methods manipulating the gradient of each loss term have become popular in Multi-Task Learning (MTL) and Continual Learning (CL) [, , ] to address conflicts between loss-specific gradients that induce negative transfer [] and catastrophic forgetting [].
Unlike weighting strategies that adjust the final update direction solely by modifying each loss-specific gradient’s magnitude,
these methods take greater flexibility and often alter the directions of loss-specific gradients.
We notice that similar conflicts between gradients also arise in the training of PINNs. First, the gradient magnitude of PDE residuals typically surpasses that of initial/boundary conditions [], causing the final update gradient to lean heavily towards the residual term. Additionally, PDE residuals often have many local minima due to the infinite number of PDE solutions without prescribed initial/boundary conditions []. Consequently, when the optimization process approaches local minima of the PDE residual, the combined update conflicts with the gradient of initial/boundary conditions, severely impeding learning progress.

Figure: Figure 1: Visualization of toy example showing the conflict between different losses during optimization.
Refer to caption: x1.png

To illustrate gradient conflicts in PINNs, we show a toy case in Fig. 1 where $\mathcal{L}_{1}$ has smaller gradients and $\mathcal{L}_{2}$ has larger gradients but multiple minima. Training with $\mathcal{L}_{1}+\lambda\mathcal{L}_{2}$ using suboptimal $\lambda$ values (e.g., $\lambda=1$) leads to gradient conflicts and the convergence to a local minima of $\mathcal{L}_{2}$.

In the following, we propose the Conflict-Free Inverse Gradients (ConFIG) method to mitigate the conflicts of loss-specific gradients during PINNs training. Our approach provides an update gradient $\bm{g}_{\text{ConFIG}}$ that reduces all loss terms to prevent the optimization from being stuck in the local minima of a specific loss term. With the ConFIG method, the optimization in Fig. 1 converges to the shared minimum for both losses. Our method is characterized by the following properties:

- •
The final update direction does not conflict with any loss-specific gradients.
- •
The projection length of the final gradient on each loss-specific gradient is uniform, ensuring that all loss terms are optimized at the same rate.
- •
The length of the final gradient is adaptively scaled based on the degree of conflict between loss-specific gradients. This prevents the optimization from stalling in the local minima of any specific loss term in high-conflict scenarios.

In addition, we introduce a momentum-based approach to expedite optimization. By leveraging momentum, we eliminate the need to compute all gradients via backpropagation at each training iteration. In contrast, we evaluate alternating loss-specific gradients, which significantly decreases the computational cost and leads to more accurate solutions within a given computational budget.

## 2 Related Work

#### Weighting Strategies for PINNs’ Training

Some intuitive weighting strategies come from the penalty method of constrained optimization problems, where higher weights are assigned to less-optimized losses [, ]. McClenny et al.  [] extended this idea by directly setting weights to sample points with soft attention masks. On the contrary, Rafael et al.  [] advocated equalizing the reduction rates of all loss terms as it guides training toward Pareto optimal solutions from an MTL perspective. Meanwhile, Wang et al.  [] addressed numerical stiffness issues by determining the weight according to the magnitude of each loss-specific gradient. Wang et al.  [] explored training procedures via the Neural Tangent Kernel (NTK) perspective, adaptively setting weights based on NTK eigenvalues. Xiang et al.  [] instead employed Gaussian probabilistic models for uncertainty quantification, estimating weights through maximum likelihood estimation. Others have pursued specialized approaches. E.g., Meer et al.  [] derived optimal weight parameters for specific problems and devised heuristics for general problems. De Wolff et al.  [] use an evolutionary multi-objective algorithm to find the trade-offs between individual loss terms in PINNs training. Shin et al.  [] conducted convergence analyses for PINNs solving linear second-order elliptic and parabolic type PDEs, employing Lipschitz regularized loss. In contrast to adaptive methods, Wight et al.  [] proposed a fixed weighting strategy for Phase field models. They artificially elevated weights for the loss term associated with initial conditions.

#### Gradient Improvement Strategies in MTL and CL

Gradient improvement strategies in MTL and CL focus on understanding and resolving conflicts between loss-specific gradients. Riemer et al.  [] and Du et al.  [] proposed using the dot product (cosine similarity) of two gradient vectors to assess whether updates will conflict with each other. Riemer et al. introduced an additional term in the loss to modify the gradient direction, maximizing the dot product of the gradients. Conversely, Du et al. chose to discard gradients of auxiliary tasks if they conflict with the main task. Another intuitive method to resolve conflicts is orthogonal projection. Mehrdad et al.  [] proposed the Orthogonal Gradient Descent (OGD) method for continual learning, projecting gradients from new tasks onto a subspace orthogonal to the previous task. Chaudhry et al.  [] proposed learning tasks in low-rank vector subspaces that are kept orthogonal to minimize interference. Yu et al.  [] introduced the PCGrad method that projects a task’s gradient onto the orthogonal plane of other gradients. Liu et al.  [] designed the IMTL-G method to ensure that the final gradient has the same projection length on all other vectors. Dong et al.  [] utilized the Singular Value Decomposition (SVD) on gradient vectors to obtain an orthogonal basis.
Javaloy et al.  [] introduces a method that jointly homogenizes gradient magnitudes and directions for MTL. Few studies in the PINN community have utilized gradient-based improvement strategies. Zhou et al.  [] introduced the PCGrad method to PINN training for reliability assessment of multi-state systems. Yao et al.  [] developed a MultiAdam method where they apply Adam optimizer for each loss term separately.

## 3 Method

### 3.1 Conflict-free Inverse Gradients Method

Generically, we consider an optimization procedure
with
a set of $m$ individual loss functions, i.e., $\{\mathcal{L}_{1},\mathcal{L}_{2},\cdots,\mathcal{L}_{m}\}$. Let $\{\bm{g}_{1},\bm{g}_{2},\cdots,\bm{g}_{m}\}$ denote the
individual gradients corresponding to each of the loss functions. A gradient-descent step with gradient $\bm{g}_{c}$ will conflict with the decrease of $\mathcal{L}_{i}$ if $\bm{g}_{i}^{\top}\bm{g}_{c}$ is negative [, ].
Thus, to ensure that all losses are decreasing simultaneously along $\bm{g}_{c}$, all $m$ components of
$[\bm{g}_{1},\bm{g}_{2},\cdots,\bm{g}_{m}]^{\top}\bm{g}_{c}$
should be positive.
This condition is fulfilled by setting
$\bm{g}_{c}=[\bm{g}_{1},\bm{g}_{2},\cdots,\bm{g}_{m}]^{-\top}\bm{w}$,
where $\bm{w}=[w_{1},w_{2},\cdots,w_{m}]$ is a vector with $m$ positive components and $M^{-\top}$ is the pseudoinverse of the transposed matrix $M^{\top}$.

Although a positive $\bm{w}$ vector guarantees a conflict-free update direction for all losses, the specific value of $w_{i}$ further influences the exact direction of $\bm{g}_{c}$. To facilitate determining $\bm{w}$, we reformulate $\bm{g}_{c}$ as $\bm{g}_{c}=k[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2}),\cdots,\mathcal{U
}(\bm{g}_{m})]^{-\top}\bm{\hat{w}}$, where $\mathcal{U}(\bm{g}_{i})=\bm{g}_{i}/(|\bm{g}_{i}|+\varepsilon)$ is a normalization operator and $k>0$. Now, $k$ controls the length of $\bm{g}_{c}$ and the ratio of $\bm{\hat{w}}$’s components corresponds to the ratio of $\bm{g}_{c}$’s projections onto each loss-specific $\bm{g}_{i}$, i.e., $|\bm{g}_{c}|\mathcal{S}_{c}(\bm{g},\bm{g}_{i})$, where $\mathcal{S}_{c}(\bm{g}_{i},\bm{g}_{j})=\bm{g}_{i}^{\top}\bm{g}_{j}/(|\bm{g}_{i
}||\bm{g}_{j}|+\varepsilon)$ is the operator for cosine similarity:

$$ $\frac{|\bm{g}_{c}|\mathcal{S}_{c}(\bm{g}_{c},\bm{g}_{i})}{|\bm{g}_{c}|\mathcal {S}_{c}(\bm{g}_{c},\bm{g}_{j})}=\frac{\mathcal{S}_{c}(\bm{g}_{c},\bm{g}_{i})}{ \mathcal{S}_{c}(\bm{g}_{c},\bm{g}_{j})}=\frac{\mathcal{S}_{c}(\bm{g}_{c},k \mathcal{U}(\bm{g}_{i}))}{\mathcal{S}_{c}(\bm{g}_{c},k\mathcal{U}(\bm{g}_{j})) }=\frac{[k\mathcal{U}(\bm{g}_{i})]^{\top}\bm{g}_{c}}{[k\mathcal{U}(\bm{g}_{j}) ]^{\top}\bm{g}_{c}}=\frac{\hat{w}_{i}}{\hat{w}_{j}}\quad\forall i,j\in[1,m].$ (1) $$

We call $\bm{\hat{w}}$ the direction weight. The projection length of $\bm{g}_{c}$ on each loss-specific gradient serves as an effective “learning rate” for each loss. Here, we choose $\hat{w}_{i}=\hat{w}_{j}\ \forall i,j\in[1,m]$ to ensure a uniform decrease rate of all losses, as it was shown to yield a weak form of Pareto optimality for multi-task learning [].

Meanwhile, we introduce an adaptive strategy for the length of $\bm{g}_{c}$ rather than directly setting a fixed value of $k$. We notice that the length of $\bm{g}_{c}$ should increase when all loss-specific gradients point nearly in the same direction since it indicates a favorable direction for optimization. Conversely, when loss-specific gradients are close to opposing each other, the magnitude of $\bm{g}_{c}$ should decrease. We realize this by rescaling the length of $\bm{g}_{c}$ to the sum of the projection lengths of each loss-specific gradient on it, i.e., $|\bm{g}_{c}|=\sum_{i=1}^{m}|\bm{g}_{i}|\mathcal{S}_{c}(\bm{g}_{i},\bm{g}_{c})$.

The procedures above are summarized in the Conflict-Free Inverse Gradients (ConFIG) operator $G$ and we correspondingly denote the final update gradient $\bm{g}_{c}$ with
$\bm{g}_{\text{ConFIG}}$:

$$ $\bm{g}_{\text{ConFIG}}=\mathcal{G}(\bm{g}_{1},\bm{g}_{1},\cdots,\bm{g}_{m}):= \left(\sum_{i=1}^{m}\bm{g}_{i}^{\top}\bm{g}_{u}\right)\bm{g}_{u},$ (2) $$

$$ $\bm{g}_{u}=\mathcal{U}\left[[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2}), \cdots,\mathcal{U}(\bm{g}_{m})]^{-\top}\mathbf{1}_{m}\right].$ (3) $$

Here, $\mathbf{1}_{m}$ is a unit vector with $m$ components. A theoretical proof of ConFIG’s convergence in convex and non-convex landscapes can be found in Appendix A.1. The ConFIG method utilizes the pseudoinverse of the gradient matrix to obtain a conflict-free direction. In Appendix A.3, we prove that such an inverse operation is always feasible as long as the dimension of parameter space is larger than the number of losses. Besides, while calculating the pseudoinverse numerically could involve additional computational cost, it is not significant compared to the cost of back-propagation for each loss term. A detailed breakdown of the computational cost can be found in Appendix A.6.

### 3.2 Two Term Losses and Positioning w.r.t. Existing Approaches

Figure: Figure 2: Sketch of PCGrad (a), IMTL-G (b), and our ConFIG (c) method with two loss terms. The PCGrad method directly sums two orthogonal components, and the IMTL-G method rescales the two vectors to the same magnitude. Our ConFIG method sums the unit vector of the orthogonal components and adjusts its magnitude with the projection length of each loss-specific gradient.
Refer to caption: x2.png

For the special case of only two loss terms, there is an equivalent form of ConFIG that does not require a pseudoinverse:

$$ $\displaystyle\mathcal{G}(\bm{g}_{1},\bm{g}_{2})=(\bm{g}_{1}^{\top}\bm{g}_{v}+ \bm{g}_{2}^{\top}\bm{g}_{v})\bm{g}_{v}$ (4) $\displaystyle\bm{g}_{v}=\mathcal{U}\left[\mathcal{U}(\mathcal{O}(\bm{g}_{1}, \bm{g}_{2}))+\mathcal{U}(\mathcal{O}(\bm{g}_{2},\bm{g}_{1}))\right]$ (5) $$

where $\mathcal{O}(\bm{g}_{1},\bm{g}_{2})=\bm{g}_{2}-\frac{\bm{g}_{1}^{\top}\bm{g}_{2
}}{|\bm{g}1|^{2}}\bm{g}_{1}$ is the orthogonality operator. It returns a vector orthogonal to $\bm{g}_{1}$ from the plane spanned by $\bm{g}_{1}$ and $\bm{g}_{2}$. The proof of equivalence is shown in Appendix A.4.

PCGrad [] and IMTL-G [] methods from multi-task learning studies also have a similar simplified form for the two-loss scenario. PCGrad projects loss-specific gradients onto the normal plane of others if their cosine similarity is negative, while IMTL-G rescales loss-specific gradients to equalize the final gradient’s projection length on each loss-specific gradient, as illustrated in Fig. 2 and 2. Our ConFIG method in Fig. 2 employs orthogonal components of each loss-specific gradient, akin to PCGrad. Meanwhile, it also ensures the same decrease rate of all losses, similar to IMTL-G. As a result, these three methods share an identical update direction but a different update magnitude in the two-loss scenario. This provides a valuable opportunity to evaluate our adaptive magnitude strategy: for two losses, the PCGrad and IMTL-G methods can be viewed as ConFIG variants with different magnitude rescaling strategies.

The above similarity between these three methods only holds for the two-loss scenario. With more losses involved, the differences between them are more evident. In fact, the inverse operation in Eq. 3 makes the ConFIG approach the only method that maintains a conflict-free direction when the number of loss terms exceeds two. Detailed discussion can be found in A.2

### 3.3 ConFIG with Momentum Acceleration

Gradient-based methods like PCGrad, IMTL-G, and the proposed ConFIG method require separate backpropagation steps to compute the gradient for each loss term. In contrast, conventional weighting strategies only require a single backpropagation for the total loss. Thus, gradient-based methods are usually $r=\sum_{i}^{m}\mathcal{T}_{b}(\mathcal{L}_{i})/\mathcal{T}_{b}\left(\sum_{i}^{
m}\mathcal{L}_{i}\right)$ times more computationally expensive than weighting methods, where $\mathcal{T}_{b}$ is the computational cost of backpropagation. To address this issue, we introduce an accelerated momentum-based variant of ConFIG: M-ConFIG. Our core idea is to leverage the momentum of the gradient for the ConFIG operation and update momentum variables in an alternating fashion to avoid backpropagating all losses in a single step. In each iteration, only a single momentum is updated with its corresponding gradient, while the others are carried over from previous steps. This
reduces the computational cost of the M-ConFIG method to $1/m$ of the ConFIG method.

Algorithm 1 details the entire procedure of M-ConFIG. It aligns with the fundamental principles of the Adam algorithm, where the first momentum averages the local gradient, and the second momentum adjusts the magnitude of each parameter. In this approach, we calculate only a single second momentum using an estimated gradient based on the output of the ConFIG operation. An alternative could involve calculating the second momentum for every loss term, similar to the MultiAdam method []. However, we found this strategy to be inaccurate and numerically unstable compared to Algorithm 1. A detailed discussion and comparison can be found in the Appendix 1.

Figure: Algorithm 1 M-ConFIG

Surprisingly, M-ConFIG not only catches up with the training speed of regular weighting strategy in PINNs experiment but typically yields an even lower average computational cost per iteration. This stems from the fact that PINN training often uses a reduced number of sampling points for boundary/initial terms, which are faster to evaluate than the residual term. Thus, $r$ is usually smaller than $m$ for PINNs training. In our experiments, we observed an average value of $r=1.67$ and a speed-up of $1.67/3\approx 0.56$ per iteration for PINNs trained with three losses using M-ConFIG.

## 4 Experiments

In this section, we employ the proposed methods for training PINNs on several challenging PDEs. We also explore the application of our method on a classical Multi-Task Learning (MTL) benchmark as an outlook.
Unless mentioned otherwise,
every result is computed via averaging three training runs
initialized with different random seeds, each using the model with the best test performance during the training.
For detailed numerical values and standard deviations of the result in each experiment, please refer to the Appendix A.9 and A.10. Training configurations and hyper-parameters of each experiment can be found in the Appendix A.11.

### 4.1 Physics Informed Neural Networks

#### Preliminaries of PINNs.

Consider the initial boundary value problem of a general PDE for a scalar function $u(\bm{x},t)$: $\mathbb{R}^{d+1}\rightarrow\mathbb{R}$ given by

$$ $\mathcal{N}[u(\bm{x},t),\bm{x},t]:=N[u(\bm{x},t),\bm{x},t]+f(\bm{x},t)=0,\quad \bm{x}\in\Omega,t\in(0,T],$ (6) $$

$$ $\mathcal{B}[u(\bm{x},t),\bm{x},t]:=B[u(\bm{x},t),\bm{x},t]+g(\bm{x},t)=0,\quad \bm{x}\in\partial\Omega,t\in(0,T],$ (7) $$

$$ $\mathcal{I}[u(\bm{x},0),\bm{x},0]:=u(\bm{x},0)+h(\bm{x},0)=0,\quad\bm{x}\in \overline{\Omega}.$ (8) $$

Here, $\Omega\subset R^{d}$ is a spatial domain with $\partial\Omega$ and $\overline{\Omega}$ denotes its boundary and closure, respectively. $N$ and $B$ are spatial-temporal differential operators, and $f$, $g$ and $h$ are source functions. To solve this initial boundary value problem, PINNs introduce a neural network $\hat{u}(\bm{x},t,\bm{\theta})$ for the target function $u(\bm{x},t)$, where $\theta$ is the weights of the neural networks. Then, spatial and temporal differential operators can be calculated efficiently with auto-differentiation of $N[\hat{u}(\bm{x},t,\bm{\theta}),\bm{x},t]$ and $B[\hat{u}(\bm{x},t,\bm{\theta}),\bm{x},t]$. Solving Eq. 6-8 turns to the training of the neural network with a loss function of

$$ $\mathcal{L}(\bm{\theta})=\underbrace{\sum_{i=1}^{n_{\mathcal{N}}}\mathcal{N}[ \hat{u}(\bm{x}_{i},t_{i},\bm{\theta}),\bm{x}_{i},t_{i}]}_{\mathcal{L}_{ \mathcal{N}}}+\underbrace{\sum_{i=1}^{n_{\mathcal{B}}}\mathcal{B}[\hat{u}(\bm{ x}_{i},t_{i},\bm{\theta}),\bm{x}_{i},t_{i}]}_{\mathcal{L}_{\mathcal{B}}}+ \underbrace{\sum_{i=1}^{n_{\mathcal{I}}}\mathcal{I}[\hat{u}(\bm{x}_{i},0,\bm{ \theta}),\bm{x}_{i},0]}_{\mathcal{L}_{\mathcal{I}}},$ (9) $$

where ($\bm{x}_{i}$,$t_{i}$) are the spatial-temporal coordinates for the data samples in the $\mathbb{R}^{d+1}$ domain, $\mathcal{L}_{\mathcal{N}}$, $\mathcal{L}_{\mathcal{B}}$ and $\mathcal{L}_{\mathcal{I}}$ are the loss functions for the PDE residual, spatial and initial boundaries, respectively. These three loss terms give us three corresponding gradients for optimization: $\bm{g}_{\mathcal{N}}$, $\bm{g}_{\mathcal{B}}$, and $\bm{g}_{\mathcal{I}}$. In the experiments, we consider four cases with three different PDEs: 1D unsteady Burgers equation, 1D unsteady Schrödinger equation, 2D Kovasznay flow (Navier-stokes equations), and 3D unsteady Beltrami flow (Navier-stokes equations). Fig. 3 shows examples of the solution domain of each PDEs. A more detailed illustration and discussion of the PDEs and corresponding solution domains can be found in Appendix A.7 and A.8, respectively.

Figure: Figure 3: Examples of PINN predictions and squared error (SE) distributions on the test PDEs.
Refer to caption: x5.png

#### Focusing on two loss terms.

Figure: Figure 4: Relative improvements of PINNs trained with two loss terms using different methods.
Refer to caption: x6.png

As detailed above, the similarity between our ConFIG method and the PCGrad/IMTL-G methods in the two-loss scenario offers a valuable opportunity to evaluate the proposed strategy for adapting the magnitude of the update. Therefore, we begin with a two-loss scenario where we introduce a new composite gradient, $\mathcal{L}_{\mathcal{BI}}$, which aggregates the contributions from both the boundary ($\mathcal{L}_{\mathcal{B}}$) and initial ($\mathcal{L}_{\mathcal{I}}$) conditions. This setup also provides better insights into the conflicting updates between PDE residuals and other loss terms.

Fig. 4 compares the performance of PINNs trained with our ConFIG method and other existing approaches. Specifically, we compare to PCGrad [] and the IMTL-G method [] from MTL studies as well as the LRA method [], MinMax [], and ReLoBRaLo [] as established weighting methods for PINNs. The accuracy metric is the MSE between the predictions and the ground truth value on the new data points sampled in the computational domain that differ from the training data points. As the central question for all these methods is how much improvement they yield in comparison to the standard training configuration with Adam, we show the relative improvement (in percent) over the Adam baseline. Thus, +50 means half, -100 twice the error of Adam, respectively. (Absolute metrics are given in the Appendix A.9)

The findings reveal that only our ConFIG and PCGrad consistently outperform the Adam baseline. In addition, the ConFIG method always performs better than PCGrad. While the IMTL-G method performs slightly better than our methods in the Kovasznay and Beltrami flow cases, it performs worse than the Adam baseline in the Schrödinger case. Fig. 5 compares the training loss of our ConFIG method with the Adam baseline approach. The results indicate that ConFIG successfully mitigates the training bias towards the PDE residual term. It achieves an improved overall
test performance by significantly decreasing the boundary/initial loss
while sacrificing the PDE residual loss slightly. This indicates that ConFIG succeeds in finding one of the many minima of the residual loss that better adheres to the boundary conditions, i.e., it finds a better overall solution for the PDE.

Figure: Figure 5: Training losses of PINNs trained with Adam baselines and ConFIG using two loss terms.
Refer to caption: x7.png

Figure: Figure 6: Relative improvements of PINNs trained with three loss terms using different methods.
Refer to caption: x12.png

Figure: Figure 7: Training losses of the Adam baseline, PCGrad, and ConFIG with three loss terms.
Refer to caption: x13.png

#### Scenarios with three loss terms.

Fig. 6 compares the performance for all three loss terms. While the general trend persists, with ConFIG and PCGrad outperforming the other methods, these cases highlight interesting differences between these two methods. As PCGrad performs better for the Burgers and Schrödinger case, while ConFIG is better for the Beltrami flow, we analyze the training losses of both methods in comparison to Adam in Fig. 7. In the Burgers and Schrödinger equation experiment, ConFIG notably reduces the loss associated with initial conditions while the loss of boundary conditions exhibits negligible change. Similarly, for PCGrad, minimal changes are observed in boundary conditions. Furthermore, owing to PCGrad’s bias towards optimizing in the direction of larger gradient magnitudes, i.e., the direction of the PDE residual, the reduction in its PDE residual term is modest, leading to a comparatively slower decrease in the loss of initial conditions compared to ConFIG. In the case of the Beltrami flow, our ConFIG method effectively reduces both boundary and initial losses, whereas the PCGrad method slightly improves boundary and initial losses. These results underscore the intricacies of PINN training, where PDE residual terms also play a pivotal role in determining the final test performance. In scenarios where the PDE residual does not significantly conflict with one of the terms, an increase in the PDE residual ultimately reduces the benefits from the improvement on the boundary/initial conditions, resulting in a better performance for PCGrad in the Burgers and Schrödinger scenario.

Figure: Figure 8: Relative improvements of the ConFIG method with different direction weights. (a) Two-loss scenario. (b) Three-loss scenario.
Refer to caption: x17.png

Figure: Figure 9: Relative improvements of the PINNs trained with different methods with the same wall time. (a) Two-loss scenario. (b) Three-loss scenario.
Refer to caption: x19.png

#### Adjusting direction weights.

In our ConFIG method, we set equal components for the direction weights $\bm{\hat{w}}=\mathbf{1}_{m}$ to ensure a uniform decrease rate across all loss terms. In the following experiments, we use the different weighting methods discussed above to calculate the components of $\bm{\hat{w}}$ and compare the results with our ConFIG method. As demonstrated in Fig. 8, only our default strategy (equal weights) and the ReLoBRaLo weights get better performance than the Adam baseline. Moreover, our equal setup consistently outperforms the ReLoBRaLo method, except for a slight inferiority in the Kovasznay flow scenario with two losses. These results further validate that the equal weighting strategy is a good choice.

#### Evaluation of Runtime Performance

While the evaluations in Fig. 4 and 6 have focused on accuracy after a given number of training epochs, a potentially more important aspect for practical applications is the accuracy per runtime. This is where the M-ConFIG method can show its full potential, as its slight approximations of the update direction come with a substantial reduction in terms of computational resources (we quantify the gains per iteration in A.6).
Fig. 9 compares the test MSE for M-ConFIG and other methods with a constant budget in terms of wall-clock time. Here, M-ConFIG outperforms all other methods, even PCGrad and the regular ConFIG method which yielded a better per-iteration accuracy above. This is apparent for all cases under consideration.

Figure: Figure 10: Test MSE of Adam baseline, ConFIG, and M-ConFIG as functions of wall time.
Refer to caption: x21.png

To shed more light on its behavior, Fig. 10 shows the test MSE of Adam, ConFIG, and M-ConFig for the most challenging scenario, the 3D unsteady Beltrami flow, with an extended training run. It shows that the advantage of M-ConFIG does not just stem from a quick late or early decrease but rather is the result of a consistently improved convergence throughout the full training. This graph additionally highlights that the regular ConFIG method, i.e., without momentum, still outperforms Adam despite its higher computational cost for each iteration.

### 4.2 Multi-task Learning

While PINNs represent a highly challenging and relevant case, we also evaluate our ConFIG method for traditional Multi-Task Learning (MTL) as an outlook. We employ the widely studied CelebA dataset [], comprising 200,000 face images annotated with 40 facial binary attributes, making it suitable for MTL with $m=40$ loss terms. Unlike PINNs, where each sub-task (loss term) carries distinct significance, and the final test loss may not reflect individual subtask performance, the CelebA experiment allows us to use the same metric to evaluate the performance for all tasks.

We compared the performance of our ConFIG method with ten popular MTL baselines. Besides PCGrad and IMTL-G from before, we also compare to Linear scalarization baseline (LS) [], Uncertainty Weighting (UW) [], Dynamic Weight Average (DWA) [], Gradient Sign Dropout (GradDrop) [], Conflict-Averse Gradient Descent (CAGrad) [], Random Loss Weighting (RLW) [], Nash bargaining solution for MTL (Nash-MTL) [], and Fast Adaptive Multitask Optimization (FAMO) []. We use two metrics to measure the performance of different methods: the mean rank metric ($MR$, lower ranks being better) [, ], represents the average rank of across all tasks. An $MR$ of 1 means consistently outperforming all others.
In addition, we consider the average F1 score ($\overline{F_{1}}$, larger is better) to measure the average performance on all tasks.

Fig. 11 presents a partial summary of the results. Our ConFIG method or M-ConFIG method emerges as the top-performing method for both $MR$ and $\overline{F_{1}}$ metric. Unlike the PINN study, we update 30 momentum variables rather than a single momentum variable for the M-ConFIG method, as indicated by the subscript ‘30.’ This adjustment is necessary because a single update step is insufficient to achieve satisfactory results in the challenging MTL experiments involving 40 tasks. Fig. 12 illustrates how the performance of the M-ConFIG method varies with different numbers of tasks and momentum-update steps in the CelebA MTL experiments. The performance of M-ConFIG tends to degrade as the number of tasks increases. This decline is attributed to longer intervals between momentum updates for each gradient, making it difficult for the momentum to accurately capture changes in the gradient. When the number of tasks becomes sufficiently large, this performance loss outweighs the benefits of M-ConFIG’s faster training speed. In our experiments, when the number of tasks reaches 10, the $\overline{F}_{1}$ score of the ConFIG method (0.635) already surpasses that of M-ConFIG (0.536) with the same training time. However, increasing the number of iterations can effectively mitigate this performance degradation. In the standard 40-task MTL scenario, increasing the number of gradient update steps per iteration to 20 allows M-ConFIG to achieve a performance level comparable to ConFIG while reducing the training time to only 56% of that required by ConFIG. Notably, alternating momentum updates can sometimes even outperform the ConFIG method with the same number of training epochs, as seen in the $n_{updates}=30$ case in the MTL experiment, as well as in the Burgers and Kovasznay cases in the two-loss PINNs experiment.

Figure: Figure 11: Test evaluation in terms of $\overline{F_{1}}$ and MR for the CelebA experiments. ConFIG and M-Config 30 yield the best performance for both metrics, each of them favoring a different metric.
Refer to caption: x22.png

Figure: Figure 12: The relative $\overline{F}_{1}$ value and training time of the M-Config method with a varying number of tasks and gradient update steps in the CelebA experiment.
Refer to caption: x23.png

#### Limitations

A potential limitation of the proposed method is the performance degradation of the M-ConFIG method as the number of losses increases. Additionally, the memory consumption required to store the momentum for each gradient grows with both the size of the parameter space and the number of loss terms. As discussed in the MTL experiment, one approach to mitigate the performance degradation is to update not just a single gradient but a stochastic subset of gradients during each iteration, albeit at the cost of an increased computational budget. Nonetheless, further investigation and improvement of the M-ConFIG method’s performance in scenarios with a large number of loss terms is a promising direction for future research, aiming to broaden the applicability of the ConFIG approach.

## 5 Conclusions

In this study, we presented ConFIG, a method designed to alleviate training conflicts between loss terms with distinct behavior. The ConFIG method achieves conflict-free training by ensuring a positive dot product between the final gradient and each loss-specific gradient. Additionally, we introduce a momentum-based approach, replacing the gradient with alternately updated momentum for highly efficient iterations. Empirically, the proposed methods have clearly shown a superior performance compared with existing training strategies for PINNs, and even outperform SOTA methods for challenging MTL scenarios.

## Acknowledgments and Disclosure of Funding

Qiang Liu acknowledges the support from the China Scholarship Council (No.202206120036
) for his Ph.D research at the Technical University of Munich.

## Appendix A Appendix

### A.1 Convergence Analysis of ConFIG method

In this section, we will give a convergence analysis for both convex (Theorem 1) and non-convex landscapes (Theorem 2) for our ConFIG method.

###### Theorem 1 .

Assume that (a) $m$ objectives $\mathcal{L}_{1},\mathcal{L}_{2}\cdots,\mathcal{L}_{m}$ are convex and differentiable; (b)The gradient $\bm{g}=\nabla_{\theta}\mathcal{L}=\sum_{i=1}^{m}\nabla_{\theta}\mathcal{L}_{i}$ is Lipschitz continuous with constant $L>0$. Then, update along ConFIG direction $\bm{g}_{\text{ConFIG}}$ with step size $\gamma\leq\frac{2}{L}$ will converge to either a location where $|\bm{g}_{\text{ConFIG}}|=0$ or the optimal solution.

###### Proof.

A Lipschitz continuous $\bm{g}$ with constant $L>0$ gives a negative semi-definite matrix $\nabla^{2}_{\theta}-LI$. If we do a quadratic expansion of $\mathcal{L}$ around $\mathcal{L}(\theta)$, we will get

$$ $\displaystyle\mathcal{L}\left(\theta^{+}\right)$ $\displaystyle\leq\mathcal{L}(\theta)+\nabla\mathcal{L}(\theta)^{T}\left(\theta ^{+}-\theta\right)+\frac{1}{2}\nabla^{2}\mathcal{L}(\theta)\left|\theta^{+}- \theta\right|^{2}$ (10) $\displaystyle\leq\mathcal{L}(\theta)+\bm{g}^{T}\left(\theta^{+}-\theta\right)+ \frac{1}{2}L\left|\theta^{+}-\theta\right|^{2}.$ $$

The $\theta^{+}$ is obtained through an update as $\theta^{+}=\theta-\gamma\bm{g}_{\text{ConFIG}}$, resulting in

$$ $\begin{aligned} \mathcal{L}\left(\theta^{+}\right)&\leq\mathcal{L}(\theta)- \gamma\bm{g}^{T}\bm{g}_{\text{ConFIG}}+\frac{1}{2}L\gamma^{2}\left|\bm{g}_{ \text{ConFIG}}\right|^{2}\\ &=\mathcal{L}(\theta)-\gamma(\sum_{i=1}^{m}\bm{g}_{i}^{T})\bm{g}_{\text{ConFIG }}+\frac{1}{2}L\gamma^{2}\left|\bm{g}_{\text{ConFIG}}\right|^{2}\\ \end{aligned}.$ (11) $$

Our $\bm{g}_{\text{ConFIG}}$ has following 2 features:

- •
An equal and positive projection on each loss-specific gradient: $\frac{\bm{g}_{i}^{\top}\bm{g}_{\text{ConFIG}}}{|\bm{g}_{i}||\bm{g}_{\text{
ConFIG}}|}=\frac{\bm{g}_{j}^{\top}\bm{g}_{\text{ConFIG}}}{|\bm{g}_{j}||\bm{g}_
{\text{ConFIG}}|}>0$, i.e., $S_{c}(\bm{g}_{i},\bm{g}_{\text{ConFIG}})=S_{c}(\bm{g}_{j},\bm{g}_{\text{ConFIG
}}):=S_{c}>0$;
- •
A magnitude equals to the sum of projection length of each loss-specific gradient on it: $|\bm{g}_{\text{ConFIG}}|=\sum_{i=1}^{m}\bm{g}_{i}^{\top}\mathcal{U}(\bm{g}_{
\text{ConFIG}})=\sum|\bm{g}_{i}|S_{c}$.

Given these two conditions, we have

$$ $\begin{aligned} (\sum_{i=1}^{m}\bm{g}_{i}^{\top})\bm{g}_{\text{ConFIG}}&=\sum_ {i=1}^{m}(\bm{g}^{\top}_{i}\bm{g}_{\text{ConFIG}})\\ &=\sum_{i=1}^{m}(|\bm{g}_{i}||\bm{g}_{\text{ConFIG}}|S_{c})\\ &=|\bm{g}_{\text{ConFIG}}|\sum_{i=1}^{m}(|\bm{g}_{i}|S_{c})\\ &=|\bm{g}_{\text{ConFIG}}|^{2}\end{aligned},$ (12) $$

turning Eq.11 into

$$ $\mathcal{L}\left(\theta^{+}\right)\leq\mathcal{L}(\theta)-\gamma|\bm{g}_{\text {ConFIG}}|^{2}+\frac{1}{2}L\gamma^{2}\left|\bm{g}_{\text{ConFIG}}\right|^{2}\\$ (13) $$

If $\gamma\leq\frac{2}{L}$, we have $\frac{1}{2}L\gamma^{2}\left|\bm{g}_{\text{ConFIG}}\right|^{2}-\gamma|\bm{g}_{
\text{ConFIG}}|^{2}\leq 0$, which finally gives $\mathcal{L}\left(\theta^{+}\right)\leq\mathcal{L}\left(\theta\right)$. Note that when $\gamma\leq\frac{2}{L}$, the equality holds and only holds when $|\bm{g}_{\text{ConFIG}}|=0$ where conflict direction doesn’t exist and optimization along any direction will results in at least one of the losses increases.
∎

###### Theorem 2 .

Assume that (a) $m$ objectives $\mathcal{L}_{1},\mathcal{L}_{2}\cdots,\mathcal{L}_{m}$ are differentiable and possibly non-convex; (b)The gradient $\bm{g}=\nabla_{\theta}\mathcal{L}=\sum_{i=1}^{m}\nabla_{\theta}\mathcal{L}_{i}$ is Lipschitz continuous with constant $L>0$. Then, update along ConFIG direction $\bm{g}_{\text{ConFIG}}$ with step size $\gamma\leq\frac{2}{L}$ will converge to either a location where $|\bm{g}_{\text{ConFIG}}|=0$ or the stationary point.

###### Proof.

Using the descent lemma, an update on $\bm{g}_{\text{ConFIG}}$ with a step size $\gamma\leq\frac{2}{L}$ gives us

$$ $\mathcal{L}\left(\theta^{k+1}\right)\leq\mathcal{L}\left(\theta^{k}\right)- \frac{\gamma}{2}|\bm{g}_{\text{ConFIG}}^{k}|^{2},$ (14) $$

where the superscript is the index of optimization iterations.

Due to $|\bm{g}_{\text{ConFIG}}|=\sum_{i=1}^{m}|\bm{g}_{i}|S_{c}$, $|\bm{g}|=|\sum_{i=1}^{m}\bm{g}_{i}|$ and the triangle inequality where $\sum_{i=1}^{m}|\bm{g}_{i}|\geq|\sum_{i=1}^{m}\bm{g}_{i}|$, we have

$$ $\begin{aligned} \mathcal{L}\left(\theta^{k+1}\right)&\leq\mathcal{L}\left( \theta^{k}\right)-\frac{\gamma}{2}|\bm{g}_{\text{ConFIG}}^{k}|^{2}\\ &=\mathcal{L}\left(\theta^{k}\right)-\frac{\gamma}{2}(\sum_{i=1}^{m}|\bm{g}_{i }^{k}|)^{2}S_{c}^{k,2}\\ &\leq\mathcal{L}\left(\theta^{k}\right)-\frac{\gamma}{2}|\sum_{i=1}^{m}\bm{g}_ {i}^{k}|^{2}S_{c}^{k,2}\\ &=\mathcal{L}\left(\theta^{k}\right)-\frac{\gamma}{2}|\bm{g}^{k}|^{2}S_{c}^{k, 2}\end{aligned}.$ (15) $$

Summing this inequality over $k=1,2,\cdots,K$ results in

$$ $\sum_{k=1}^{K}|\bm{g}^{k}|^{2}\leq\frac{2}{\gamma}\sum_{k=1}^{K}\frac{\mathcal {L}\left(\theta^{k}\right)-\mathcal{L}\left(\theta^{k+1}\right)}{S_{c}^{k,2}}.$ (16) $$

By defining $\min_{1\leq k\leq K}(S_{c}^{k})=\alpha>0$, we have

$$ $\sum_{k=1}^{K}|\bm{g}^{k}|^{2}\leq\frac{2}{\gamma\alpha^{2}}\left[\mathcal{L} \left(\theta^{0}\right)-\mathcal{L}\left(\theta^{K+1}\right)\right].$ (17) $$

Finally, averaging Eq.17 on each side leads to

$$ $\min_{1\leq k\leq K}|\bm{g}^{k}|^{2}\leq\frac{1}{K}\sum_{k=1}^{K}|\bm{g}^{k}|^ {2}\leq\frac{2}{\gamma\alpha^{2}K}\left[\mathcal{L}\left(\theta^{0}\right)- \mathcal{L}\left(\theta^{K+1}\right)\right].$ (18) $$

As $K\rightarrow\infty$, either the $|\bm{g}_{\text{ConFIG}}|=0$ is obtained and the optimization stops, or the last term in the inequality goes 0 0, implying that the minimal gradient norm also goes to zero, i.e., a stationary point.
∎

###### Theorem 1 .

###### Proof.

###### Theorem 2 .

###### Proof.

### A.2 Comparison between ConFIG, PCGrad and IMTL-G

Figure: Algorithm 2 PCGrad method

#### PCGrad method.

Algorithm 2 outlines the complete algorithm of the PCGrad method. The core idea is to use $\mathcal{O}(\bm{g}_{2},\bm{g}_{1})$ to replace $\bm{g}_{1}$ when $\bm{g}_{1}$ conflicts with $\bm{g}_{2}$ i.e., $\bm{g}_{1}^{\top}\bm{g}_{2}<0$. Since $\mathcal{O}(\bm{g}_{2},\bm{g}_{1})^{\top}\bm{g}_{2}=0$, the conflict is resolved. In the simple case of two loss terms, this results in the final gradient $\bm{g}_{\text{PCGrad}}=\mathcal{O}(\bm{g}_{1},\bm{g}_{2})+\mathcal{O}(\bm{g}_{
2},\bm{g}_{1})$. This final update gradient does not conflict with all loss-specific gradients, as $\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{2})+\mathcal{O}(\bm{g}_{2},\bm{g}_{1})
\right]^{\top}\bm{g}_{1}=\mathcal{O}(\bm{g}_{2},\bm{g}_{1})^{\top}\bm{g}_{1}=|
\bm{g}_{1}|^{2}-\frac{(\bm{g}_{1}^{\top}\bm{g}_{2})^{2}}{|\bm{g}_{2}^{2}|}\geq
|\bm{g}_{1}|^{2}-\frac{|\bm{g}_{1}|^{2}|\bm{g}_{2}|^{2}}{|\bm{g}_{2}^{2}|}=0$.

However, the PCGrad method can not guarantee a conflict-free update when there are more loss-specific gradients. To illustrate this, let us consider the case of three vectors, $\bm{g}_{1},\bm{g}_{2},\bm{g}_{3}$, where each vector conflicts with the other. According to the PCGrad method, we first replace $\bm{g}_{1}$ with $\mathcal{O}(\bm{g}_{2},\bm{g}_{1})$, i.e., $\hat{\bm{g}}_{1}=\mathcal{O}(\bm{g}_{2},\bm{g}_{1})$ so that $\hat{\bm{g}}_{1}^{\top}\bm{g}_{2}=0$. If the updated $\hat{\bm{g}}_{1}$ is still conflict with $\bm{g}_{3}$, we will then further modify $\hat{\bm{g}}_{1}$ as $\hat{\bm{g}}_{1}=\mathcal{O}(\bm{g}_{3},\hat{\bm{g}}_{1})$. However, now we can only guarantee that $\hat{\bm{g}}_{1}^{\top}\bm{g}_{3}=0$, but we can not guarantee that $\hat{\bm{g}}_{1}$ is still not conflicting with $\bm{g}_{2}$.

The PCGrad method introduces random selection to mitigate this drawback, as shown in Algorithm 2, but this does not fundamentally solve the problem. Fig. 13 illustrate a simple case for the failure of the PCGrad method where $\bm{g}_{1}=[1.0,0,0.1]$, $\bm{g}_{2}=[-0.5,\sqrt{3}/2,0.1]$, and $\bm{g}_{3}=[-0.5,-\sqrt{3}/2,0.1]$. The final update vector for the PCGrad method is $\bm{g}_{\text{PCGrad}}=[-0.351,-0.203,0.658]$, which conflicts with $\bm{g}_{1}$ while the update direction of our ConFIG method is $\bm{g}_{\text{ConFIG}}=[0,0,0.3]$, which is not conflict with any loss-specific gradients.

Figure: Figure 13: A simple failure model for PCGrad method.
Refer to caption: x25.png

Another feature of the PCGrad method is that the projection length of the final update gradient $\bm{g}_{\text{PCGrad}}$ on each loss-specific gradient is not equal, i.e., $\bm{g}_{\text{PCGrad}}^{\top}\mathcal{U}(\bm{g}_{i})\neq\bm{g}_{\text{PCGrad}}
^{\top}\mathcal{U}(\bm{g}_{j})$, and $\bm{g}_{\text{PCGrad}}$ is usually biased to the loss-specific gradient with higher magnitude. We can illustrate this point with two gradient examples:

$$ $\displaystyle\begin{split}\bm{g}_{\text{PCGrad}}^{\top}\mathcal{U}(\bm{g}_{1}) &=(\mathcal{O}(\bm{g}_{1},\bm{g}_{2})+\mathcal{O}(\bm{g}_{2},\bm{g}_{1}))^{ \top}\frac{\bm{g}_{1}}{|\bm{g}_{1}|}\\ &=\mathcal{O}(\bm{g}_{2},\bm{g}_{1})^{\top}\frac{\bm{g}_{1}}{|\bm{g}_{1}|}\\ &=\left(\bm{g}_{1}-\frac{\bm{g}_{1}^{\top}\bm{g}_{2}}{|\bm{g}_{2}|^{2}}\bm{g}_ {2}\right)^{\top}\frac{\bm{g}_{1}}{|\bm{g}_{1}|}\\ &=|\bm{g}_{1}|-\frac{(\bm{g}_{1}^{\top}\bm{g}_{2})^{2}}{|\bm{g}_{2}|^{2}|\bm{g }_{1}|}\\ &=|\bm{g}_{1}|[1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})].\end{split}$ (19) $$

Similarly,

$$ $\bm{g}_{\text{PCGrad}}^{\top}\mathcal{U}(\bm{g}_{2})=|\bm{g}_{2}|[1-\mathcal{S }_{c}^{2}(\bm{g}_{1},\bm{g}_{2})].$ (20) $$

Thus,

$$ $\frac{\bm{g}_{\text{PCGrad}}^{\top}\mathcal{U}(\bm{g}_{1})}{\bm{g}_{\text{ PCGrad}}^{\top}\mathcal{U}(\bm{g}_{2})}=\frac{|\bm{g}_{1}|}{|\bm{g}_{2}|}$ (21) $$

A similar conclusion also holds for the case with more loss-specific gradients. The essence here is the magnitude of the orthogonal operator which is proportional to the magnitude of loss-specific gradients:

$$ $\displaystyle\begin{split}|\mathcal{O}(\bm{g}_{i},\bm{g}_{j})|&=\sqrt{(\bm{g}_ {j}-\frac{\bm{g}_{i}^{\top}\bm{g}_{j}}{|\bm{g}_{i}|^{2}}\bm{g}_{i})^{2}}\\ &=\sqrt{(|\bm{g}_{j}|^{2}+\frac{(\bm{g}_{i}^{\top}\bm{g}_{j})^{2}}{|\bm{g}_{i} |^{2}}-2\frac{\bm{g}_{i}^{\top}\bm{g}_{j}}{|\bm{g}_{i}|^{2}}\bm{g}_{i}^{\top} \bm{g}_{j})}\\ &=\sqrt{|\bm{g}_{j}|^{2}-\frac{(\bm{g}_{i}^{\top}\bm{g}_{j})^{2}}{|\bm{g}_{i}| ^{2}}}\\ &=\sqrt{|\bm{g}_{j}|^{2}-|\bm{g}_{j}|^{2}\mathcal{S}_{c}^{2}(\bm{g}_{i},\bm{g} _{j})}\\ &=|\bm{g}_{j}|\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{i},\bm{g}_{j})}.\end{split}$ (22) $$

When summing all these orthogonal values together, the final gradient will be biased to the $\mathcal{O}(\bm{g}_{i},\bm{g}_{j})$ with a larger gradient. Meanwhile, since $\mathcal{O}(\bm{g}_{i},\bm{g}_{j})$ is biased towards $\bm{g}_{j}$ rather than $\bm{g}_{i}$(due to $\mathcal{S}_{c}(\bm{g}_{i},\mathcal{O}(\bm{g}_{i},\bm{g}_{j}))=0$), the final gradient will favor to the loss-specific gradient with larger magnitude.

#### IMTL-G method.

In contrast to the PCGrad method the IMTL-G method is designed to guarantee an equal projection length, i.e.,

$$ $\bm{g}_{\text{IMTL-G}}\cdot\mathcal{U}(g_{i})^{\top}=\bm{g}_{\text{IMTL-G}} \cdot\mathcal{U}(g_{j})^{\top}$ (23) $$

To achieve this goal, the IMTL-G method rescales the magnitude of each loss-specific gradient with calculated weights:

$$ $\bm{g}_{\text{IMTL-G}}=\sum_{i=1}^{m}\alpha_{i}\bm{g}_{i}$ (24) $$

$$ $\begin{cases}\left[\alpha_{2},\alpha_{3}\cdots\alpha_{m}\right]=\bm{g}_{1} \cdot\bm{U}^{\top}\cdot\left(\bm{D}\cdot\bm{U}^{\top}\right)^{-1}\\ \alpha_{1}=1-\sum_{i=2}^{m}\alpha_{i}\\ \bm{U}^{\top}=[\mathcal{U}(\bm{g}_{1}^{\top})-\mathcal{U}(\bm{g}_{2}^{\top}), \mathcal{U}(\bm{g}_{1}^{\top})-\mathcal{U}(\bm{g}_{3}^{\top}),\cdots,\mathcal{ U}(\bm{g}_{1}^{\top})-\mathcal{U}(\bm{g}_{m}^{\top})]\\ \bm{D}^{\top}=[\bm{g}_{1}^{\top}-\bm{g}_{2}^{\top},\bm{g}_{1}^{\top}-\bm{g}_{3 }^{\top},\cdots,\bm{g}_{1}^{\top}-\bm{g}_{m}^{\top}]\end{cases}$ (25) $$

For the two gradient scenario, this will give $[\alpha_{1},\alpha_{2}]=[|\bm{g}_{2}|/(|\bm{g}_{1}|+|\bm{g}_{2}|),|\bm{g}_{1}|
/(|\bm{g}_{1}|+|\bm{g}_{2}|)]$, resulting in a same magnitude for two loss-specific gradients as $(|\bm{g}_{1}||\bm{g}_{2}|)/(|\bm{g}_{1}|+|\bm{g}_{2}|)$. The final gradient vector doesn’t conflict with $\bm{g}_{1}$ and $\bm{g}_{2}$, as

$$ $\displaystyle\begin{split}\bm{g}_{\text{IMTL-G}}^{\top}\mathcal{(}U)(\bm{g}_{1 })&=\frac{|\bm{g}_{1}||\bm{g}_{2}|}{|\bm{g}_{1}|+|\bm{g}_{2}|}(\frac{\bm{g}_{1 }}{|\bm{g}_{1}|}+\frac{\bm{g}_{2}}{|\bm{g}_{2}|})^{\top}\frac{\bm{g}_{1}}{|\bm {g}_{1}|}\\ &=\frac{|\bm{g}_{1}||\bm{g}_{2}|}{|\bm{g}_{1}|+|\bm{g}_{2}|}(1+\mathcal{S}_{c} (\bm{g}_{1},\bm{g}_{2}))\\ &\geq 0\end{split}$ (26) $$

As mentioned in Appendix. 3.2, the direction of the final update gradient for IMTL-G and our ConFIG is the same when there are only two vectors, but their magnitudes differ. For the Adam optimizer, the gradient’s absolute magnitude is unimportant since it ultimately rescales the gradient elements. The crucial aspect is how the gradient changes in both direction and magnitude. When the gradient changes rapidly, Adam’s corresponding learning rate will decrease, as discussed in Appendix. A.5. Since the direction of the final gradient in IMTL-G and ConFIG is the same, how their magnitudes change ultimately affects their performance with the Adam optimizer. The magnitude of the final gradient with two gradients are $2\sqrt{\left[1+\mathcal{S}_{c}(\bm{g}_{1}),\bm{g}_{2})\right]/2}(|\bm{g}_{1}||
\bm{g}_{2}|)/(|\bm{g}_{1}|+|\bm{g}_{2}|)$ and $\sqrt{\left[1+\mathcal{S}_{c}(\bm{g}_{1}),\bm{g}_{2})\right]/2}(|\bm{g}_{1}|+|
\bm{g}_{2}|)$ for IMTL-G and ConFIG method, respectively. Thus,

$$ $\frac{|\bm{g}_{\text{IMTL-G}}|}{|\bm{g}_{\text{ConFIG}}|}=\frac{2(|\bm{g}_{1}| |\bm{g}_{2}|)/(|\bm{g}_{1}|+|\bm{g}_{2}|)}{|\bm{g}_{1}|+|\bm{g}_{2}|}$ (27) $$

A significant difference arises when the magnitudes of $\bm{g}_{1}$ and $\bm{g}_{2}$ are different. For example, if $\bm{g}_{1}>>\bm{g}_{2}$, $(|\bm{g}_{1}||\bm{g}_{2}|)/(|\bm{g}_{1}|+|\bm{g}_{2}|)\sim\bm{g}_{2}$ while $|\bm{g}_{1}|+|\bm{g}_{2}|\sim\bm{g}_{1}$. This means that the magnitude of $\bm{g}_{\text{IMTL-G}}$ tracks the changes in the magnitude of the smaller vector, whereas our ConFIG method tracks the magnitude of the larger vector. This causes the difference between IMTL-G and our ConFIG methods in two loss-term vector situations.

Figure: Figure 14: A simple failure model for IMTL-G method.
Refer to caption: x26.png

Another major difference between IMTL-G and our ConFIG methods occurs when more gradient terms are involved. The coefficient $\alpha_{i}$ of IMTL-G method might have negative values since Eq. 25 only guarantees that $\bm{g}_{\text{IMTL-G}}\cdot\mathcal{U}(g_{i})^{\top}=\bm{g}_{\text{IMTL-G}}
\cdot\mathcal{U}(g_{j})^{\top}$ but not $\bm{g}_{\text{IMTL-G}}\cdot\mathcal{U}(g_{i})^{\top}=\bm{g}_{\text{IMTL-G}}
\cdot\mathcal{U}(g_{j})^{\top}\geq 0$. It is easy to find such a situation where the negative coefficient will result in a conflicting update. Fig. 14 illustrates such an example where $\bm{g}_{1}=[0.0412,0.4295,0.9394]$, $\bm{g}_{2}=[0.3571,0.5491,0.1414]$, and $[0.9823,0.9361,0.0552]$. Although both our ConFIG and IMTL-G ensure $\bm{g}_{\text{IMTL-G}}\cdot\mathcal{U}(g_{i})^{\top}=\bm{g}_{\text{IMTL-G}}
\cdot\mathcal{U}(g_{j})^{\top}$, they are in exactly opposite direction. The obtained update direction for our ConFIG method is $\bm{g}_{\text{ConFIG}}=[1.5844,0.4850,1.4005]$ and the cosine similarity between $\bm{g}_{\text{ConFIG}}$ and each loss-specific gradient is 0.7086 while IMTL-G method results in a final update gradient of $\bm{g}_{\text{IMTL-G}}=[-0.7429,-0.2274,-0.6566]$ with a cosine similarity of -0.7086.

### A.3 Existence of Solutions for ConFIG

In Appendix A.2, we illustrate instances where the PCGrad and IMTL-G methods fail. This raises the question of whether ConFIG likewise exhibits failure modes that could impede its performance.

Our ConFIG method is equivalent to solving the following system of linear equations:

$$ $[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2}),\cdots,\mathcal{U}(\bm{g}_{m} )]^{\top}\bm{x}=\mathbf{1}_{m},$ (28) $$

where the solution $\bm{x}$ is the conflict-free gradient that should be obtained. Our ConFIG method will fail if Eq. 28 has no solution, i.e., if a conflict-free direction in the parameter space does not exist. If we use $\bm{A}$ and $\bm{b}$ to denote $[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2}),\cdots,\mathcal{U}(\bm{g}_{m}
)]^{\top}$ and $\mathbf{1}_{m}$, respectively,
Eq. 28 will not have a solution if $R(\bm{A})<R[(\bm{A}|\bm{b})]$ where $R$ is the rank of a matrix and $(\bm{A}|\bm{b})$ is the augmented matrix of $\bm{A}$ and $\bm{b}$. Meanwhile, since $R(\bm{b})$=1, and $R(\bm{A})\leq R[(\bm{A}|\bm{b})]\leq R(\bm{A})+R(\bm{b})$, i.e., $R(\bm{A})\leq R[(\bm{A}|\bm{b})]\leq R(\bm{A})+1$, $R[(\bm{A}|\bm{b})]$ can only be either $R(\bm{A})$ or $R(\bm{A})+1$. Thus, Eq. 28 doesn’t have solutions when $R[(\bm{A}|\bm{b})]=R(\bm{A})+1$.

Assume that each loss-specific gradient has $k$ elements, then the shape of $\bm{A}$ and $\bm{b}$ is $m\times k$ and $m\times(k+1)$, respectively. Thus, we have:

- •
If $\bm{A}$ is full rank:
–
If $m>k$: $R(\bm{A})=k$;
*
if $(\bm{A}|\bm{b})$ is full rank: $R[(\bm{A}|\bm{b})]=k+1$. Eq. 28 doesn’t have solutions.
*
if $(\bm{A}|\bm{b})$ is not full rank:
$R[(\bm{A}|\bm{b})]=k$. Eq. 28 has solutions.
–
If $m\leq k$: $R(\bm{A})=m$.
Since $R[(\bm{A}|\bm{b})]\leq min(m,k+1)=m$, thus $R[(\bm{A}|\bm{b})]\neq R(\bm{A})+1=m+1$. Eq. 28 has solutions.
- •
If $\bm{A}$ is not full rank, i.e., $R(\bm{A})=p<min(m,k)$,
we will always have $p+1\leq min(m,k+1)$. Thus, there is no guarantee that $R[(\bm{A}|\bm{b})]\neq m+1$. Thus Eq. 28 doesn’t have solutions if $R[(\bm{A}|\bm{b})]=p+1$, and it has solutions if $R[(\bm{A}|\bm{b})]=p$.

In practice, the number of parameters in a neural network is always much larger than the number of loss terms, i.e., $k>>m$. In this case, a full rank $\bm{A}$ means $R(\bm{A})=m$, i,e, all the loss-specific gradients are linearly independent. This is true in most situations because if the loss-specific gradients were not linearly independent, we would have $\bm{g}_{j}=\sum_{i=1,i\neq j}^{m}\alpha_{i}\bm{g}_{i}$, where $\alpha_{i}$ is the coefficient. Since $\bm{g}_{i}=\partial\mathcal{L}_{i}/\partial\theta$, this would also imply $\mathcal{L}_{j}=\sum_{i=1,i\neq j}^{m}\alpha_{i}\mathcal{L}_{i}$. However, the relationship between loss functions is usually not linear in practice.

To summarize, during the training procedure, we almost always have $k>>m$ and a full rank $\bm{A}$, ensuring the existence of a conflict-free direction. It is also worth pointing out that our ConFIG method can still provide a good direction if a conflict-free direction does not exist since the Moore–Penrose pseudoinverse will offer a least squares solution for Eq. 28, making it a “conflict-free” as possible.

### A.4 Proof of the Equivalence

To prove the equivalence between Eq. 2, 3 and Eq. 4, 5 is equivalent to prove that
$\mathcal{U}\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{2})\right]+\mathcal{U}\left[
\mathcal{O}(\bm{g}_{2},\bm{g}_{1})\right]=k[\mathcal{U}(\bm{g}_{1}),\mathcal{U
}(\bm{g}_{2})]^{-\top}\mathbf{1}_{m}$ where k>0.
Thus, we can first calculate

$$ $\displaystyle\begin{split}[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2})]^{ \top}\left[\mathcal{U}\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{2})\right]+\mathcal {U}\left[\mathcal{O}(\bm{g}_{2},\bm{g}_{1})\right]\right]&=\begin{bmatrix} \mathcal{U}(\bm{g}_{1})^{\top}\left[\mathcal{U}\left[\mathcal{O}(\bm{g}_{1}, \bm{g}_{2})\right]+\mathcal{U}\left[\mathcal{O}(\bm{g}_{2},\bm{g}_{1})\right] \right]\\ \mathcal{U}(\bm{g}_{2})^{\top}\left[\mathcal{U}\left[\mathcal{O}(\bm{g}_{1}, \bm{g}_{2})\right]+\mathcal{U}\left[\mathcal{O}(\bm{g}_{2},\bm{g}_{1})\right] \right]\end{bmatrix}\\ &=\begin{bmatrix}\mathcal{U}(\bm{g}_{1})^{\top}\mathcal{U}\left[\mathcal{O}( \bm{g}_{2},\bm{g}_{1})\right]\\ \mathcal{U}(\bm{g}_{2})^{\top}\mathcal{U}\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{ 2})\right]\end{bmatrix}.\end{split}$ (29) $$

Then, from Eq. 19 we can obtain

$$ $\mathcal{U}(\bm{g}_{1})^{\top}\mathcal{O}(\bm{g}_{2},\bm{g}_{1})=|\bm{g}_{1}|[ 1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})].$ (30) $$

Thus

$$ $\mathcal{U}(\bm{g}_{1})^{\top}\mathcal{U}\left[\mathcal{O}(\bm{g}_{2},\bm{g}_{ 1})\right]=\frac{|\bm{g}_{1}|[1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})]}{| \mathcal{O}(\bm{g}_{2},\bm{g}_{1})|}.$ (31) $$

From Eq. 22 we can get

$$ $|\mathcal{O}(\bm{g}_{2},\bm{g}_{1})|=|\bm{g}_{1}|\sqrt{1-\mathcal{S}_{c}^{2}( \bm{g}_{1},\bm{g}_{2})}.$ (32) $$

Put Eq. 32 back to Eq. 31, we get

$$ $\mathcal{U}(\bm{g}_{1})^{\top}\mathcal{U}\left[\mathcal{O}(\bm{g}_{2},\bm{g}_{ 1})\right]=\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})}.$ (33) $$

Similarly, we can get

$$ $\mathcal{U}(\bm{g}_{2})^{\top}\mathcal{U}\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{ 2})\right]=\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})}.$ (34) $$

Put Eq. 33 and Eq. 34 back to Eq. 29

$$ $[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2})]^{\top}\left[\mathcal{U}\left [\mathcal{O}(\bm{g}_{1},\bm{g}_{2})\right]+\mathcal{U}\left[\mathcal{O}(\bm{g} _{2},\bm{g}_{1})\right]\right]=[\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_ {2})},\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{1},\bm{g}_{2})}]^{\top},$ (35) $$

which is

$$ $\mathcal{U}\left[\mathcal{O}(\bm{g}_{1},\bm{g}_{2})\right]+\mathcal{U}\left[ \mathcal{O}(\bm{g}_{2},\bm{g}_{1})\right]=\sqrt{1-\mathcal{S}_{c}^{2}(\bm{g}_{ 1},\bm{g}_{2})}[\mathcal{U}(\bm{g}_{1}),\mathcal{U}(\bm{g}_{2})]^{-\top} \mathbf{1}_{m}.$ (36) $$

### A.5 M-Config and Momentum Strategies

In the current study, we use the Adam optimizer to train baseline PINNs, as shown in Algorithm 3. The green line in the algorithm can be replaced by other methods like PCGrad or the IMTL-G method. The Adam optimizer uses the first momentum $m$ to average the recent update direction and the second momentum $v$ to rescale the learning rate of each parameter. If we want to use the momentum to replace the gradient for the ConFIG update, an intuitive approach could be to calculate the first and second momentum for each loss-specific gradient and use the final rescaled momentum to calculate the input for the ConFIG update, as shown in "MA-ConFIG" in Algorithm 4. However, this approach could result in several issues:

- •
Inappropriate learning rate. Ideally, the Adam method will rescale each element of the update gradient to 1 ($\bm{m}\sim\bm{g}$, $\bm{v}\sim\bm{g}^{2}$, $\bm{m}/\sqrt{\bm{v}}\sim 1$). If one of the elements of the update gradient repeatedly oscillates between positive and negative, then $m$ becomes smaller while $v$ becomes larger, resulting in a small update step (learning rate) for the corresponding parameter. However, Adam’s neat adaptive learning rate adjustment feature will be destroyed if we rescale the momentum before applying the ConFIG operator. This is because ConFIG will alter each parameter’s learning rate again by changing the update gradient’s direction and magnitude.
- •
Numerical instability. The current study uses singular value decomposition (SVD) to calculate the pseudoinverse. Our experience shows that rescaling the momentum makes the input matrix for SVD easily ill-conditioned or has too many repeated singular values, leading to training failure.

Tab. 1 compares the performance of M-ConFIG and MA-ConFIG methods, where M-ConFIG is always stable and significantly better than the MA-ConFIG method.

Note that the M-ConFIG method should not be implemented in combination with Adam, as it is intended to replace Adam’s calculation of momentum.
Hence, our M-ConFIG implementation uses an SGD optimizer.

Figure: Algorithm 3 ConFIG update with Adam optimizer

Figure: Algorithm 4 MA-ConFIG

**Table 1: Test results of different momentum configurations. "MA-ConFIG" refers to the strategy with the second momentum inside the ConFIG operation, and "NaN" means the training failed due to numerical stability.**
|  |  | M-ConFIG | MA-ConFIG |
| --- | --- | --- | --- |
| Burgers | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $(1.277\pm 0.035)\times 10^{-4}$ | $(1.549\pm 0.362)\times 10^{-3}$ |
| Schrödinger | $(4.292\pm 1.863)\times 10^{-4}$ | $(2.625\pm 0.087)\times 10^{-1}$ |  |
| Kovasznay | $(9.777\pm 0.347)\times 10^{-9}$ | NaN |  |
| Beltrami | $(7.949\pm 0.384)\times 10^{-5}$ | $(6.658\pm 0.435)\times 10^{-3}$ |  |
| Burgers | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ | $(1.296\pm 0.013)\times 10^{-4}$ | $(8.511\pm 7.835)\times 10^{-3}$ |
| Schrödinger | $(1.522\pm 0.581)\times 10^{-3}$ | NaN |  |
| Beltrami | $(9.103\pm 1.831)\times 10^{-5}$ | $(5.783\pm 0.405)\times 10^{-3}$ |  |

### A.6 Computational Cost

To evaluate runtime performance, Fig. 15 compares the relative wall time of all methods w.r.t. the Adam baseline, highlighting the significant advantage of the M-ConFIG on the training cost. The average training cost of M-ConFIG is 0.712$\times$ that of the Adam baseline for the two-term case, and 0.557$\times$ for the three-term case.

It is also worth noting that the time cost of the ConFIG method is comparable to other gradient-based methods, indicating that the pseudoinverse operation of the ConFIG method does not add significant computational overhead. This study uses the PyTorch implementation, which utilizes singular value decomposition (SVD) to calculate the pseudoinverse. Although the exact computational cost depends on the specific numerical algorithm, the general time complexity of SVD is $\mathcal{O}(nm^{2})$ [] where $n$ is the number of gradient elements (i.e., the number of neural network’s parameter) and $m$ is the number of gradients in the current study ($n>m$).

This is favorable since the number of network parameters is usually much larger than the number of loss terms, and the time complexity of SVD scales linearly with the former.

Figure: Figure 15: Relative wall time of different methods w.r.t. the Adam baseline for one training iteration.
Refer to caption: x27.png

### A.7 PDE Details

#### 1D Burgers equation.

Burgers equation is a non-linear and non-trivial model equation describing shock formations. Considering a spatial-temporal field $u(x,t)$, the Burgers equation in one space dimension is

$$ $\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}=\nu\frac{\partial ^{2}u}{\partial x^{2}},$ (37) $$

where $\nu$ is the viscosity and set as $\nu=0.01/\pi$ in the current study. The spatial-temporal domain is $t\in[0,1]$ and $x\in[-1.0,1.0]$ with the corresponding initial and boundary conditions of

$$ $\begin{cases}u(x,0)=-\sin({\pi x})\\ u(+1.0,t)=u(-1.0,t)=0\\ \end{cases}.$ (38) $$

No analytical solution is available for Burgers equation with these given boundary conditions. In the current study, we use numerical solutions from PhiFlow [] as the ground truth solution to evaluate the PINNs’ performance.

#### 1D Schrödinger equation.

1D Schrödinger equation describes the evaluation of a complex field $h(x,t)=u(x,t)+iv(x,t)$ in one space dimension:

$$ $i\frac{\partial h}{\partial t}+\frac{1}{2}\frac{\partial^{2}h}{\partial x^{2}} +|h|^{2}h=0,$ (39) $$

$x\in[-5.0,5.0]$, $t\in[0,\pi/2]$. It follows a periodic boundary condition and a initial condition of

$$ $h(0,x)=2\operatorname{sech}(x),\\$ (40) $$

Similarly, no analytical solution is available for the 1D Schrödinger equation with given boundary conditions. We use the numerical solution provided by deepXDE [] to evaluate the performance of trained PINNs.

#### 2D Kovasznay flow and 3D Beltrami flow.

Kovasznay and Beltrami flow are special solutions for Navier-Stokes equations which describe the fluid flow:

$$ $\displaystyle\frac{\partial\bm{u}}{\partial t}+(\bm{u}\cdot\nabla)\bm{u}$ $\displaystyle=-\nabla p+\frac{1}{Re}\nabla^{2}\bm{u}$ (41) $\displaystyle\nabla\cdot\bm{u}$ $\displaystyle=0$ $$

where $\bm{u}$ is the velocity vector, $p$ is the pressure and $Re$ is the Reynolds number. Kovasznay flow is a steady case in two space dimensions where the transient term $\partial\bm{u}/\partial t$ equals 0. The analytical solution of the velocity $\bm{u}(x,y)=[u_{x}(x,y),u_{y}(x,y)]$ and pressure $p(x,y)$ is [, ]

$$ $\begin{cases}u_{x}(x,y)&=1-e^{\lambda x}\cos(2\pi y),\\ u_{y}(x,y)&=\frac{\lambda}{2\pi}e^{\lambda x}\sin(2\pi y),\\ p(x,y)&=\frac{1}{2}\left(1-e^{2\lambda x}\right),\end{cases}$ (42) $$

where $\lambda=\frac{1}{2\nu}-\sqrt{\frac{1}{4\nu^{2}}+4\pi^{2}}$,$\quad\nu=\frac{1}{\operatorname{Re}}=\frac{1}{40}$, $x\in[-0.5,1]$ and $y\in[-0.5,1.5]$. Meanwhile, Beltrami flow is an unsteady case in three space dimensions where the velocity $\bm{u}(x,y,z,t)=[u_{x}(x,y,z,t),u_{y}(x,y,z,t),u_{z}(x,y,z,t)]$ and pressure $p(x,y,z,t)$ follows the analytical solution of []

$$ $\begin{cases}u_{x}(x,y,z,t)=&-a\left[e^{ax}\sin(ay+dz)+e^{az}\cos(ax+dy)\right ]e^{-d^{2}t}\\ u_{y}(x,y,z,t)=&-a\left[e^{ay}\sin(az+dx)+e^{ax}\cos(ay+dz)\right]e^{-d^{2}t} \\ u_{z}(x,y,z,t)=&-a\left[e^{az}\sin(ax+dy)+e^{ay}\cos(az+dx)\right]e^{-d^{2}t} \\ p(x,y,z,t)=&-\frac{1}{2}a^{2}\left[e^{2ax}+e^{2ay}+e^{2az}\right.\\ &+2\sin(ax+dy)\cos(az+dx)e^{a(y+z)}\\ &+2\sin(ay+dz)\cos(ax+dy)e^{a(z+x)}\\ &\left.+2\sin(az+dx)\cos(ay+dz)e^{a(x+y)}\right]e^{-2d^{2}t}\end{cases},$ (43) $$

where $a=d=1$ when $Re=1$ in our configuration. The simulation domain size is $x\in[-1,1]$, $y\in[-1,1]$, $z\in[-1,1]$, and $t\in[0,1]$. The boundary conditions of Kovasznay and Beltrami flow are Dirichlet type, which follows the values of analytical solutions. For Beltrami flow, the initial condition also follows the values of the analytical solutions.

### A.8 PINN-based Example Solutions

This section provides examples of the solution domain for each PINN case. Figures 16, 18, 20, and 22 show the mean predictions of the networks compared to the ground truth, the standard deviation, and the squared error distribution for each PINN case. Additionally, Figures 17, 19, and 21 offer a detailed comparison of a sample line in the solution domain for the Burgers equation, Schrödinger equation, and Kovasznay flow case.

Figure: Figure 16: The solution domain of Burgers Equation with corresponding standard deviation($\sigma$) and square error(SE) of the ConFIG method’s prediction.
Refer to caption: x32.png

Figure: Figure 17: The distribution of $u$ in the solution domain of Burgers equation.
Refer to caption: x33.png

Figure: Figure 18: The solution domain of Schrödinger Equation with corresponding standard deviation($\sigma$) and square error(SE) of the ConFIG method’s prediction.
Refer to caption: x34.png

Figure: Figure 19: The distribution of $|h|$ in the solution domain of Schrödinger equation.
Refer to caption: x35.png

Figure: Figure 20: The solution domain of Kovasznay Flow with corresponding standard deviation($\sigma$) and square error(SE) of the ConFIG method’s prediction.
Refer to caption: x36.png

Figure: Figure 21: The distribution of $u_{x}$ in the solution domain of Kovasznay Flow.
Refer to caption: x37.png

Figure: Figure 22: The solution domain of Beltrami Flow when $t=0.5$ with corresponding standard deviation($\sigma$) and square error(SE) of the ConFIG method’s prediction.
Refer to caption: x38.png

### A.9 PINN Evaluations

The exact numerical values and corresponding standard deviations of the PINN experiments are presented in this section. Tables 2 through 5 display the values obtained with the same number of training epochs as depicted in Fig. 4 and Fig. 6. Additionally, Tables 6 through 9 showcase the values obtained with identical wall time, corresponding to the data presented in Fig. 9.

**Table 2: Test MSE of PINNs trained for Burgers equation. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $1.484\pm 0.061$ |  |
| PCGrad | $1.344\pm 0.019$ | $\mathbf{1.279\pm 0.008}$ |
| IMTL-G | $1.339\pm 0.024$ | $22.478\pm 15.008$ |
| MinMax | $1.889\pm 0.143$ | $1.582\pm 0.051$ |
| ReLoBRaLo | $1.419\pm 0.053$ | $1.402\pm 0.034$ |
| LRA | $353.796\pm 114.972$ | $444.603\pm 16.695$ |
| ConFIG | $1.308\pm 0.008$ | $1.291\pm 0.039$ |
| M-ConFIG | $\mathbf{1.277\pm 0.035}$ | $1.296\pm 0.013$ |

**Table 3: Test MSE of PINNs trained for Schrödinger equation. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $3.383\pm 1.178$ |  |
| PCGrad | $1.621\pm 0.547$ | $\mathbf{0.931\pm 0.028}$ |
| IMTL-G | $7.891\pm 3.008$ | $2504.282\pm 588.560$ |
| MinMax | $45.255\pm 1.535$ | $45.068\pm 5.292$ |
| ReLoBRaLo | $3.603\pm 2.165$ | $2.756\pm 0.098$ |
| ConFIG | $\mathbf{0.643\pm 0.227}$ | $1.455\pm 0.455$ |
| M-ConFIG | $4.292\pm 1.863$ | $15.217\pm 5.807$ |

**Table 4: Test MSE of PINNs trained for Kovasznay flow. All values are scaled with $10^{-7}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ |
| --- | --- |
| Adam | $1.044\pm 0.405$ |
| PCGrad | $0.799\pm 0.083$ |
| IMTL-G | $\mathbf{0.096\pm 0.012}$ |
| MinMax | $7.743\pm 1.197$ |
| ReLoBRaLo | $1.148\pm 0.359$ |
| LRA | $3.595\pm 3.582$ |
| ConFIG | $0.126\pm 0.048$ |
| M-ConFIG | $0.098\pm 0.003$ |

**Table 5: Test MSE of PINNs trained for Beltrami flow. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $1.112\pm 0.119$ |  |
| PCGrad | $1.037\pm 0.119$ | $0.757\pm 0.082$ |
| IMTL-G | $0.568\pm 0.078$ | $0.932\pm 0.228$ |
| MinMax | $1.851\pm 0.100$ | $2.215\pm 0.192$ |
| ReLoBRaLo | $1.197\pm 0.029$ | $1.028\pm 0.107$ |
| LRA | $0.708\pm 0.047$ | $0.843\pm 0.146$ |
| ConFIG | $\mathbf{0.617\pm 0.112}$ | $\mathbf{0.571\pm 0.090}$ |
| M-ConFIG | $0.795\pm 0.038$ | $0.910\pm 0.183$ |

**Table 6: Test MSE of PINNs trained for Burgers equation. All values are collected after the same wall time and scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $1.515\pm 0.079$ | $1.546\pm 0.100$ |
| PCGrad | $1.476\pm 0.022$ | $1.300\pm 0.023$ |
| IMTL-G | $1.356\pm 0.046$ | $82.446\pm 76.016$ |
| MinMax | $3.058\pm 0.379$ | $2.030\pm 0.153$ |
| ReLoBRaLo | $1.449\pm 0.033$ | $1.492\pm 0.099$ |
| LRA | $376.449\pm 100.599$ | $470.108\pm 28.172$ |
| ConFIG | $1.330\pm 0.015$ | $1.665\pm 0.186$ |
| M-ConFIG | $\mathbf{1.277\pm 0.035}$ | $\mathbf{1.296\pm 0.013}$ |

**Table 7: Test MSE of PINNs trained for Schrödinger equation. All values are collected after the same wall time and scaled with $10^{-3}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $2.169\pm 0.584$ | $5.832\pm 1.421$ |
| PCGrad | $1.505\pm 0.559$ | $2.271\pm 0.205$ |
| IMTL-G | $8.844\pm 4.263$ | $291.037\pm 3.682$ |
| MinMax | $28.195\pm 1.616$ | $49.635\pm 2.888$ |
| ReLoBRaLo | $2.369\pm 0.829$ | $15.230\pm 7.691$ |
| LRA | $316.549\pm 8.696$ | $315.606\pm 8.969$ |
| ConFIG | $0.586\pm 0.256$ | $4.636\pm 0.932$ |
| M-ConFIG | $\mathbf{0.429\pm 0.186}$ | $\mathbf{1.522\pm 0.581}$ |

**Table 8: Test MSE of PINNs trained for Kovasznay flow. All values are scaled with $10^{-7}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ |
| --- | --- |
| Adam | $4.037\pm 1.078$ |
| PCGrad | $2.723\pm 0.421$ |
| IMTL-G | $0.419\pm 0.058$ |
| MinMax | $261.628\pm 43.303$ |
| ReLoBRaLo | $3.291\pm 0.847$ |
| LRA | $13.554\pm 13.773$ |
| ConFIG | $0.631\pm 0.194$ |
| M-ConFIG | $\mathbf{0.098\pm 0.003}$ |

**Table 9: Test MSE of PINNs trained for Beltrami flow. All values are collected after the same wall time and scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| Adam | $2.318\pm 0.270$ | $3.648\pm 0.244$ |
| PCGrad | $2.206\pm 0.180$ | $2.351\pm 0.212$ |
| IMTL-G | $1.054\pm 0.156$ | $2.194\pm 0.465$ |
| MinMax | $5.422\pm 0.300$ | $15.754\pm 1.546$ |
| ReLoBRaLo | $2.605\pm 0.056$ | $2.769\pm 0.202$ |
| LRA | $1.194\pm 0.139$ | $2.259\pm 0.812$ |
| ConFIG | $1.130\pm 0.138$ | $1.451\pm 0.181$ |
| M-ConFIG | $\mathbf{0.795\pm 0.038}$ | $\mathbf{0.910\pm 0.183}$ |

**Table 10: Test MSE of PINNs trained for Burgers equation with the ConFIG method using different direction weights. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| MinMax | $2.444\pm 0.312$ | $45.068\pm 5.292$ |
| ReLoBRaLo | $1.310\pm 0.007$ | $1.315\pm 0.048$ |
| LRA | $449.658\pm 42.392$ | $657.258\pm 45.487$ |
| Equal | $\mathbf{1.308\pm 0.008}$ | $\mathbf{1.291\pm 0.039}$ |

**Table 11: Test MSE of PINNs trained for Schrodinger equation with the ConFIG method using different direction weights. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| MinMax | $2.444\pm 0.312$ | $3.733\pm 1.351$ |
| ReLoBRaLo | $1.048\pm 0.404$ | $2.829\pm 1.444$ |
| LRA | $25.417\pm 6.768$ | $3118.378\pm 45.723$ |
| Equal | $\mathbf{6.429\pm 2.269}$ | $\mathbf{1.455\pm 0.455}$ |

**Table 12: Test MSE of PINNs trained for Kovasznay flow with the ConFIG method using different direction weights. All values are scaled with $10^{-7}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ |
| --- | --- |
| MinMax | $7.487\pm 2.545$ |
| ReLoBRaLo | $\mathbf{0.104\pm 0.008}$ |
| LRA | $(9.047\pm 2.052)\times 10^{5}$ |
| Equal | $0.126\pm 0.048$ |

**Table 13: Test MSE of PINNs trained for Beltrami flow with the ConFIG method using different direction weights. All values are scaled with $10^{-4}$.**
|  | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{BI}}]$ | $[\mathcal{L}_{\mathcal{N}},\mathcal{L}_{\mathcal{B}},\mathcal{L}_{\mathcal{I}}]$ |
| --- | --- | --- |
| MinMax | $1.903\pm 0.271$ | $1.597\pm 0.039$ |
| ReLoBRaLo | $0.658\pm 0.157$ | $0.575\pm 0.059$ |
| LRA | $155.617\pm 55.853$ | $15.754\pm 5.049$ |
| Equal | $\mathbf{0.617\pm 0.112}$ | $\mathbf{0.571\pm 0.090}$ |

### A.10 Results of CelebA MTL Experiments

Table 14- 19 presents the numerical values and corresponding standard deviations for the CelebA MTL experiments, aligning with the results in Fig. 11 and Fig. 12. The best performance is determined by selecting the best average performance across different tasks during training, which is also the value shown in Fig. 11. The average performance is calculated based on the results from the last 5 epochs, during which most methods have converged. For both best and average performance, our ConFIG method surpasses all other methods in $MR$ metrics. For the performance in $\overline{F_{1}}$, it tied for first with the FAMO method in best performance and ranked second in the average performance.

**Table 14: $MR$ performance of different methods in CelebA MTL experiments**
|  | Best performance | Average performance |
| --- | --- | --- |
| PCGrad | $6.425\pm 0.595$ | $5.475\pm 0.575$ |
| IMTL-G | $6.058\pm 0.671$ | $5.475\pm 0.125$ |
| FAMO | $6.233\pm 0.085$ | $4.875\pm 0.100$ |
| NASHMTL | $6.158\pm 0.242$ | $5.612\pm 1.237$ |
| RLW | $6.300\pm 0.602$ | $4.850\pm 1.050$ |
| CAGrad | $6.767\pm 0.274$ | $7.387\pm 1.062$ |
| GRADDROP | $6.733\pm 0.309$ | $7.375\pm 0.000$ |
| DWA | $6.908\pm 0.450$ | $8.500\pm 0.000$ |
| LS | $7.383\pm 0.112$ | $8.675\pm 0.075$ |
| UW | $7.692\pm 0.051$ | $8.238\pm 0.438$ |
| M-ConFIG 30 | $5.833\pm 0.342$ | $7.175\pm 0.525$ |
| ConFIG | $\mathbf{5.508\pm 0.392}$ | $\mathbf{4.362\pm 0.812}$ |

**Table 15: $\overline{F_{1}}$ performance of different methods in CelebA MTL experiments**
|  | Best performance | Average performance |
| --- | --- | --- |
| PCGrad | $0.681\pm 0.003$ | $0.665\pm 0.007$ |
| IMTL-G | $0.680\pm 0.003$ | $0.660\pm 0.003$ |
| FAMO | $0.686\pm 0.004$ | $\mathbf{0.673\pm 0.002}$ |
| NASHMTL | $0.683\pm 0.006$ | $0.642\pm 0.026$ |
| RLW | $0.680\pm 0.002$ | $0.663\pm 0.006$ |
| CAGrad | $0.672\pm 0.010$ | $0.649\pm 0.008$ |
| GRADDROP | $0.661\pm 0.005$ | $0.644\pm 0.010$ |
| DWA | $0.672\pm 0.003$ | $0.644\pm 0.004$ |
| LS | $0.676\pm 0.004$ | $0.639\pm 0.004$ |
| UW | $0.675\pm 0.009$ | $0.641\pm 0.006$ |
| M-ConFIG 30 | $\mathbf{0.694\pm 0.005}$ | $0.655\pm 0.006$ |
| ConFIG | $0.686\pm 0.006$ | $0.671\pm 0.003$ |

**Table 16: The $\overline{F}_{1}$ performance of the M-Config and ConFIG method with different numbers of tasks in the CelebA experiment.**
| $n_{\text{updates}}$ | Best performance | Average performance |  |  |
| --- | --- | --- | --- | --- |
| ConFIG | M-ConFIG | ConFIG | M-ConFIG |  |
| 5 | $0.697\pm 0.007$ | $0.664\pm 0.005$ | $0.645\pm 0.006$ | $0.638\pm 0.003$ |
| 10 | $0.695\pm 0.006$ | $0.536\pm 0.057$ | $0.668\pm 0.013$ | $0.488\pm 0.055$ |
| 20 | $0.684\pm 0.008$ | $0.455\pm 0.032$ | $0.660\pm 0.007$ | $0.408\pm 0.015$ |
| 30 | $0.667\pm 0.006$ | $0.429\pm 0.023$ | $0.648\pm 0.006$ | $0.383\pm 0.025$ |
| 40 | $0.686\pm 0.006$ | $0.423\pm 0.026$ | $0.671\pm 0.003$ | $0.383\pm 0.037$ |

**Table 17: The training time of the M-Config and ConFIG method with different numbers of tasks in the CelebA experiment.**
| $n_{\text{tasks}}$ | ConFIG | M-ConFIG |
| --- | --- | --- |
| 5 | $180.333\pm 0.471$ | $67.436\pm 0.267$ |
| 10 | $288.000\pm 0.816$ | $73.588\pm 1.262$ |
| 20 | $561.000\pm 0.816$ | $84.252\pm 0.919$ |
| 30 | $861.000\pm 0.816$ | $110.454\pm 0.487$ |
| 40 | $1179.000\pm 2.449$ | $167.423\pm 0.427$ |

**Table 18: The $\overline{F}_{1}$ performance of the M-Config and ConFIG method with different numbers of gradient updates in the CelebA experiment (40 tasks).**
| $n_{\text{updates}}$ | Best performance | Average performance |  |  |
| --- | --- | --- | --- | --- |
| ConFIG | M-ConFIG | ConFIG | M-ConFIG |  |
| 1 | $0.686\pm 0.006$ | $0.423\pm 0.026$ | $0.671\pm 0.003$ | $0.383\pm 0.037$ |
| 5 | $0.458\pm 0.006$ | $0.406\pm 0.008$ |  |  |
| 10 | $0.570\pm 0.049$ | $0.517\pm 0.049$ |  |  |
| 20 | $0.681\pm 0.006$ | $0.668\pm 0.008$ |  |  |
| 30 | $0.694\pm 0.005$ | $0.678\pm 0.006$ |  |  |
| 40 | $0.682\pm 0.007$ | $0.654\pm 0.015$ |  |  |

**Table 19: The training time of the M-Config and ConFIG method with different numbers of gradient updates in the CelebA experiment (40 tasks).**
| $n_{\text{updates}}$ | ConFIG | M-ConFIG |
| --- | --- | --- |
| 1 | $2090.618\pm 5.627$ | $310.000\pm 0.000$ |
| 5 | $492.667\pm 1.247$ |  |
| 10 | $723.000\pm 0.816$ |  |
| 20 | $1185.667\pm 6.018$ |  |
| 30 | $1647.667\pm 7.587$ |  |
| 40 | $2084.667\pm 24.635$ |  |

### A.11 Training Details

#### PINNs.

The training of PINNs in the current research follows the established conventions. The neural networks are fully connected with 4 hidden layers and 50 channels per layer. The activation function is the Tanh function, and all weights are initialized with Xavier initialization []. Data points are sampled using Latin-hypercube sampling and updated in each iteration. The extended training run case shown in Fig. 10 uses a constant learning rate of $10^{-4}$. All other cases follow a cosine decay strategy with the initial and final learning rate of $10^{-3}$ and $10^{-4}$, respectively. We also add a learning rate warm-up of 100 epochs for each training. All the methods except M-ConFIG use Adam optimizer. The hyper-parameters of the Adam optimizer are set as $\beta_{1}$=0.9, $\beta_{2}=0.999$, and $\varepsilon=10^{-8}$, respectively. The number of data points and training epochs for each case are listed in Tab. 20.

**Table 20: The number of data points and training epochs for PINNs’ experiments.**
|  | Number of data points | Epochs |  |  |
| --- | --- | --- | --- | --- |
|  | $n_{\mathcal{N}}$ | $n_{\mathcal{B}}$ | $n_{\mathcal{I}}$ |  |
| Burgers equation | 10000 | 250 | 250 | $3\times 10^{4}$ |
| Schrödinger equation | 20000 | 500 | 500 | $10^{5}$ |
| Kovasznay flow | 20000 | 1000 |  |  |
| Beltrami flow | 25000 | 5000 | 5000 |  |

#### MTL.

Our experiments are based on the official test code of the FAMO method, and our ConFIG method implemented in the corresponding framework.
For the details of the configurations, please refer to the original FAMO paper [] and its official repository: [https://github.com/Cranial-XIX/FAMO](https://github.com/Cranial-XIX/FAMO) (MIT License).

#### Experiments compute resources.

All the experiments in this study were conducted using an NVIDIA RTX A5000 GPU with 24 GB of memory. Each PINN experiment discussed can be completed within a few hours on any modern GPU with more than 4 GB of memory. For the CelebA MTL test, a GPU with more than 12 GB of memory is required, and a single training run may take 1-2 days, depending on GPU performance.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: NA
soundness: 4
strengths: 1.  The experiments thoroughly evaluate the proposed method from various aspects and for different problems.
    - The effectiveness (in terms of accuracy) of both ConFIG and Momentum(M)-ConFIG algorithms is demonstrated compared to other gradient adjustment methods, loss weighting methods for PINNs, and Adam.
    - While M-ConFIG falls short of the original ConFIG, its performance in computation-constrained scenarios is well shown.
2. The motivating problem from PINNs is a significant obstacle to wider adoption of them, and the proposed algorithm paves the way for their application to more problems in Physics.
3. The paper is well-written and accompanied by a reasonable amount of experiments, maths, and pseudo-code.
summary: Motivated by the discrepancy between the residual and data loss terms in Physics-Informed Neural Networks (PINNs), the authors propose a "conflict-free" gradient update for multi-task learning, such that the resulting optimization direction minimizes all individual loss terms, avoiding local minima induced by one loss term. Such optimization step is calculated by ensuring positive cosine similarity between the update direction and the individual gradients of each loss term and (in most cases) a uniform decay rate for all losses. The authors also propose a momentum-based algorithm to alleviate the computational costs.
weaknesses: 1. Although the experiments are already extensive, the paper can benefit from experiments with other PDEs (e.g. KS, Turbulence) and boundary conditions (e.g. Neumann) from the studied loss weighting baselines.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: In addition to the points raised in weaknesses, it would be good if the authors can address the following:
- It seems like this approach has been primarily studied in the context of forward problems. Have you performed any studies on inverse problems as well?
- Additionally, I'm curious how the more recent pre-print of Jacobian Descent for Multi-Objective Optimization (Quinton & Rey) ties in, and compares against your work. Again, they compare against several MTL aggregators and propose a new strategy that claims to avoid conflicts with any of the individual gradients.
soundness: 4
strengths: - The paper addresses the important issue of conflicting directions in gradient updates in the training of PINNs. The proposed approach is novel and seems to combine the best of other MTL methods that have been applied in the context of PINNs such as PCGrad and IMTL-G. The proposed formulation seems well-formulated and is theoretically sound. Further, the introduction of M-ConFIG as a method to accelerate training while avoiding the evaluation of all the individual gradients at every step is a great practical contribution. Additionally, the appendices on this aspect are also thorough and discuss the reason as to why using Adam with ConFIG wouldn't work.

- The experiments also reveal a better performance than other baselines across different PDE benchmarks. Moreover, the method also performs well when applied to the standard MTL CelebA benchmark revealing broader applicability. The experiments also showed better convergence when compared against computational time of other optimization schemes further improving the applicability.
summary: This paper proposes a novel method (ConFIG) to deal with multi-task problems, and specifically focus on the task of training physics-informed neural networks (PINNs). The main motivation behind the method is that multiple loss functions in their formulation (PDE residuals, boundary / initial conditions) often conflict and lead to suboptimal optimizations. To overcome this limitations, ConFIG ensures that the update direction is conflict free by maintaining a positive-dot product between the final direction and the individual loss-specific gradients. Additionally, the paper introduces a momentum-based version (M-ConFIG) which reduces computational efficiency while maintaining training efficiency. The proposed method is contrasted against other loss aggregation schemes on several PDE problems and a multi-task learning benchmark, and demonstrates improved performance over existing methods.
weaknesses: - The experiments that are peformed in the paper compare against other MTL aggregation schemes for the proposed benchmark problems. However, there are several challenging benchmark problems for PINNs that have been established in other works (such as PirateNets (Wang et al.), PINNacle (Hao et al.),  Expert's Guide to PINNs (Wang et al.)). Although, they do use other enhancements such as an improved network architecture / improved encodings / imposing causality, it would be interesting to see if these enhancements would further improve on the current state of the art results.

- The hyperparameter tuning and ablation studies in the paper could be more extensive. For instance, the authors claim that the learning rate was fixed to $10^{-4}$ for the long-training or the use of a $10^{-3} \to 10^{-4}$ decay. But they don't seem to study the influence of this chosen rate on the final relative L2 error that has been obtained. It would strengthen the position of the paper to study this aspect as well.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: In addition to what I mentioned in weaknesses, I would like to know how this type of models can be compared to traditional methods to improve PINN training typically involves adjusting the weights for PDE residuals and loss terms for initial and boundary conditions in terms of convergence and performance.
soundness: 3
strengths: - The optimization of different loss functions in PINNs is critical, as it can influence whether PINNs succeed or fail in solving forward problems. The exploration of Multi-Task Learning (MTL) strategies for improving PINNs training is innovative and may inspire future methodologies.
- The problem setup and method are clearly explained, making the paper accessible and informative. The theoretical background is well-supported by proofs and discussions.
- The experimental design is good, but it needs some adjustments.
- Given the difficulty of training PINNs due to conflicting gradient directions and numerical issues, the ConFIG method can be a good direction for solving these problems.
summary: This paper addresses the challenge of gradient conflicts in Physics-Informed Neural Networks (PINNs) and the tendency of optimization to get stuck in the local minima of specific loss terms. The authors highlight that the gradient magnitude of PDE residuals typically dominates that of initial and boundary condition losses, leading to biased final updates. Their proposed ConFIG method leverages a momentum-based approach that reduces computational costs by alternating the backpropagation of loss-specific gradients, ensuring conflict-free updates and improved convergence.
weaknesses: - The examples explored in this study are relatively simple and can be solved with standard methods like Adam. It would be insightful to evaluate the proposed approach on more complex, chaotic PDEs, such as the Kuramoto-Sivashinsky equation, which poses significant challenges due to its non-linear and chaotic behavior.
- The study omits comparisons with certain MTL strategies, such as FAMO, which could provide further context to the method’s effectiveness in PINNs. Additionally, using only the CelebA dataset to demonstrate performance in MTL may not be comprehensive enough.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. What is the computational complexity of the pseudoinverse operation in ConFIG, and how does it scale with larger problem sizes?

2. It would be better if the authors could test ConFIG on high-dimensional or multi-scale PDE problems, as seen in benchmarks like PINNacle? If so, how does it perform?

3. The authors are expected to provide more insight into how sensitive ConFIG is to hyperparameters like learning rate and batch size? 

4. The authors are expected to evaluat ConFIG’s scalability and efficiency for large-scale industrial applications.
soundness: 3
strengths: The paper makes a notable contribution to the field of PINN training by addressing a fundamental challenge with an original and technically sound approach.

Originality: The ConFIG method introduces a novel solution to gradient conflict resolution in PINNs. It effectively adapts ideas from multi-task learning (MTL) and continual learning (CL) to address gradient conflicts, which is relatively unexplored in the PINN domain.

Quality: The paper demonstrates strong technical quality with rigorous mathematical proofs and comprehensive experimental validation across multiple benchmark problems. While the method’s effectiveness is well-supported, additional comparisons with state-of-the-art methods and scalability analysis would further bolster its quality.

Clarity: The paper is generally well-organized and clear, though certain sections, particularly those with heavy mathematical notation, could be simplified for broader accessibility. 

Significance: The method addresses a critical challenge in PINN training, with broad implications for fields like fluid dynamics, electromagnetics, and beyond.
summary: The paper introduces a novel method, ConFIG (Conflict-Free Inverse Gradients), aimed at addressing gradient conflicts that arise during the training of Physics-Informed Neural Networks (PINNs). The primary challenge in training PINNs lies in the conflicting gradient directions from multiple loss terms, particularly those associated with initial/boundary conditions and the physics equations, which can impede learning and optimization.  The paper proposes the ConFIG method, which ensures that the final update gradient has a positive dot product with each loss-specific gradient, eliminating conflicts and enabling consistent optimization across all loss terms. The method adjusts gradient magnitudes dynamically based on conflict levels and maintains uniform optimization rates for all loss terms. The method provides a robust and conflict-free optimization framework for PINNs, and its application extends to multi-task learning, showcasing improved performance over existing methods.
weaknesses: The paper would significantly increase its rigor and relevance by including more comprehensive comparisons, and the detailed comments are 
1. Limited Benchmarking: The evaluation is narrow, focusing on standard PDEs like Burgers equations, without addressing more complex, real-world PDE challenges (e.g., multi-scale phenomena, high-dimensionality). It would be helpful if the authors could broaden the evaluation to include more diverse and complex PDEs, as seen in benchmarks like PINNacle (Hao et al., 2024), to demonstrate the robustness of ConFIG across a wider range of problems.

2. Comparison with Advanced Methods: The paper primarily compares ConFIG to basic methods (e.g., PCGrad, IMTL-G), without considering recent, more sophisticated approaches in PINN optimization. The authors are expect to include comparisons with advanced methods.

3. Scalability and Efficiency: The scalability of ConFIG, especially regarding the computational cost is not fully explored. It would be better if the authors could provide a detailed analysis of runtime and memory efficiency, particularly for large-scale or high-dimensional problems, to validate ConFIG’s scalability.

4. Ablation and Sensitivity Studies: There’s a lack of ablation studies to assess the impact of key hyperparameters like learning rate and gradient scaling. The authors could perform more ablation studies to better understand the effect of different components and configurations on ConFIG’s performance.

Meta-review: This paper proposes **ConFIG** for multi-objective optimization, ensuring positive cosine similarity between update directions and consistent optimization rates for all loss terms. The authors also introduce a momentum-based variant to accelerate convergence. The paper presents extensive experiments, focusing in particular on physics-informed neural networks (PINNs), but also demonstrating improvements on a general multi-task benchmark. All reviewers gave very positive feedback. Although the approach still cannot solve certain challenging chaotic PDE problems, this requires innovations beyond the scope of this work. Given strong reviews and positive results across extensive experiments, I recommend accepting this paper as spotlight.

Decision: Accept (Spotlight)