Title: Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scientific Machine Learning

Physics-informed Neural Networks

Partial Differential Equations

Second-order Optimization

Gradient Conflicts
Abstract: Physics-informed neural networks (PINNs) have shown significant promise in computational science and engineering, yet they often face optimization challenges and limited accuracy. In this work, we identify directional gradient conflicts during PINN training as a critical bottleneck. We introduce a novel gradient alignment score to systematically diagnose this issue through both theoretical analysis and empirical experiments.
Building on these insights, we show that (quasi) second-order optimization methods inherently mitigate gradient conflicts, thereby consistently outperforming the widely used Adam optimizer. Among them, we highlight the effectiveness of SOAP \cite{vyas2024soap} by establishing its connection to Newton’s method.
Empirically, SOAP achieves state-of-the-art results on 10 challenging PDE benchmarks, including the first successful application of PINNs to turbulent flows at Reynolds numbers up to 10,000. It yields 2–10x accuracy improvements over existing methods while maintaining computational scalability, advancing the frontier of neural PDE solvers for real-world, multi-scale physical systems. All code and datasets used in this work are publicly available at:  
\url{https://github.com/PredictiveIntelligenceLab/jaxpi/tree/pirate}.
\end{abstract}


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper thoroughly investigates the issue of gradient alignment that arises during training of PINNs, which arise when combining the physics-informed loss (residuals of the PDE) and the supervised one (boundary conditions/ initial condition/ data). The authors propose a novel gradient alignment score, inspired from the cosine similarity, to quantify these conflicts and demonstrate how and why quasi-second-order optimization alleviate them. The paper combines the theoretical analysis with extensive numerical validation across benchmark PDEs.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper is in a very good format. In particular:\n* The paper is very well written and provides a clean and complete understanding of the problem studied, both numerically and theoretically.\n* The underlying problem is a critical issue in PINNs for years, and the analysis proposed by the authors gives a clear and easy understanding of the problem\n* The alignment score introduced gives a very intuitive understanding of the empirical results, while carrying interesting theoretical advancements\n\nHowever, there are some factors that are concerning.\n* The problem of loss balancing in PINNs has been already studied and solved in several literature.\n* The authors strongly recommends the usage of SOAP, but the theoretical backbone suggests that a pure second-order method would be the optimal choice. Comparison with methods as Natural Gradient Descent, Levemberg-Marquardt or other quasi-Newton approaches would bring value to the choice of SOAP.\n* The improvements in the relative $L^2$ error shown in Table 2 are in general unimpressive. The usage of second-order methods can boost the accuracy of PINNs of several orders of magnitude, even when the underlying network is small (making it feasible to adopt such methods).\n* According to the experimental setup described in the appendix, PINNs trained in the numerical evaluation use random weight factorization, random Fourier features, a PirateNet architecture and gradient norm-based loss weighting. Some of these methods are implicitly addressing the gradient alignment issue, making it harder to understand the standalone impact of SOAP.\n\nQUESTIONS:\nI have the following questions:\n* Second-order training for PINNs can be very effective even with small (approx. 2000 parameters) and vanilla NN architectures. How does the SOAP optimizer perform in this scenario? Why not including such results to highlight the effect of SOAP alone?\n* Based on the theoretical findings, I would have expected SOAP to reach much higher accuracy. Quasi-Newton methods as L-BFGS can at times face struggle when the hessian matrix they approximate is ill-conditioned (as often the case with PINNs). Can it be that the SOAP pre-conditioner is sensible to the approximation of singular hessians?\n\nLIMITATIONS:\nThe authors clearly discuss the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the gradient alignment of multiple objectives in Physics-informed neural networks (PINNs). Training of PINNs faces optimization challenges due to the composite nature of the loss function, which consists of competing objectives (e.g. the initial and boundary conditions and a PDE residual). \nBy introducing a new alignment score, defined as \n\n$$\n\\mathcal{A}(v_1,v_2,\\dots,v_n) := 2 \\left|\\left| \\frac{\\sum_{i=1}^n \\frac{v_i}{\\||v_i\\||}}{n} \\right|\\right|^2 -1,\n$$\nwhich generalizes the cosine similarity measure to multiple vectors, the authors identify the misalignment of gradient directions of competing objectives as a critical obstacle. \nBuilding on this result, the authors show that (quasi) second-order optimization methods can mitigate these gradient conflicts resulting in superior performance over classically used methods, including Adam, and that a recently proposed optimizer SOAP performs best among the tested methods.\nFinally, this paper shows empirically that PINNs trained with SOAP are able to learn challenging PDEs, including turbulent flows, which was not possible before with PINNs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper is overall clearly written and easy to follow. \n- This paper introduces a novel alignment score, which shows that the update directions of multiple objectives (they are not strictly gradient directions anymore due to the diagonal preconditioning) in Adam are misaligned, while this is less the case in quasi second-order methods, such as Kron, Muon and SOAP. This alignment score also correlates strongly with the relative L2-error achieved by each optimizer across multiple benchmark problems.\n- I am not an expert in PINNs, but the empirical results that the authors achieve in this work with the SOAP optimizer appear to be quite impressive, which advances the state-of-the-art of what PINNs can achieve.\n- The benchmarks that the authors provide can serve as a valuable reference for future work on PINNs. \n\nWeaknesses\n- The novelty of this paper seems a bit limited, given that SOAP has been proposed already and that gradient conflicts appear to be a known problem in training PINNs given that there is previous work which propose specifically designed optimizers (e.g. ConFiG) to address this problem (even though the overall performance is below SOAP).\n- The provided explanation of SOAP's superior performance is not very convincing to me. Proposition 3 only addresses the inter-step alignment (which is equivalent to cosine similarity), not the intra-step alignment, which seems to be the more critical point regarding gradient conflicts. Furthermore, Proposition 3 only provides a statement regarding the maximal (local) step size that can be taken to maintain a certain gradient alignment between consecutive steps, but does neither explain why higher alignment scores between consecutive steps is necessarily better nor why a larger step size is more desired.\n\nQUESTIONS:\n- I am a little bit puzzled regarding the inter-step alignment score of SOAP. The plots show that the alignment score is almost or exactly 1.0, which would imply that the trajectory of the iterates is almost straight. This seems a bit unlikely to me, given that all the methods use mini-batches. Could the authors perhaps clarify on this? \n\n- Can the authors also elaborate on why a very high inter-step alignment score close to 1 is desired? (which SOAP seems to achieve, see my question above.) Clearly having a negative or close-to zero inter-step alignment score is undesirable, but given the highly non-convex landscape, I would not expect the trajectory to be very straight. For instance, [1] also studies the directionality of trajectories in neural networks (although not in PINNs), but argue that a cosine similarity score of consecutive gradients close to 1 would be undesired. \n\n- I had a closer look at Figure 5 and noticed that the relative L2-error of SOAP dominates all other methods except for the Korteweg-De Vries problem, which aligns with the higher intra-step alignment score. Here Muon has the lowest relative L2-error for three quarter of the training trajectory, while having comparably low alignment scores, even lower than Adam and Kron (after initial 20% of steps). Do the authors have any explanation for this? \n\n- Given that all these methods have different computational costs, can the authors also provide figures for relative L2-error vs. wall-clock time? \n\n[1] Singh, Sidak Pal, et al. \"The directionality of optimization trajectories in neural networks.\" The Thirteenth International Conference on Learning Representations. 2025.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the gradient alignment problem of PINNs, which belongs to an important topic that has been researched extensively. The authors locate the issue both theoretically and empirically. They propose to use quasi second-order methods, especially SOAP, to resolve gradient conflicts, which shows superior accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The decomposition of two types of gradient conflicts and inter-step/intra-step gradient alignment.\n- Theoretical explanation of the model's dynamics.\n- Good performance on a range of datasets\n\nWeaknesses:\n\n1. Some datasets are not very challenging, on which Adam has already achieved sub 1% error rates. I recommend testing SOAP on some hard datasets such as Poisson2d-MS and Wave2d-CG from PINNacle, where the boundary conditions are more difficult.\n2. Didn't show why second-order methods can help with inter-step gradient alignment.\n3. The method-wise contribution is limited by borrowing SOAP, which affects the novelty of this work.\n4. One key contribution as claimed is \"We reveal that SOAP [1] can be viewed as an efficient approximation of the Newton preconditioner.\" But this is revealed by SOAP, not this work.\n\nQUESTIONS:\n1. The authors only focus on type II of the gradient conflicts, \"where gradients have similar magnitudes but opposing directions.\" It would be interesting to see type I of such during PINN's training, particularly while second-order methods are applied, and whether the two types are orthogonal. By orthogonality I mean whether different learning rate annealing algorithms can work with SOAP.\n2. As the authors listed, there are many quasi second-order approaches. Why does specifically SOAP (and sometimes Muon) perform better than others? Insights or analyses are appreciated.\n\nLIMITATIONS:\nThey are mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper investigates a key optimization challenge in PINNs, where conflicting gradients from different loss terms hinder effective training. The authors propose a novel gradient alignment score, distinguishing between intra-step (within a step) and inter-step (across steps) misalignments. They reveal that first-order optimizers like Adam suffer from poor alignment, especially during early training.

To address this, they propose a quasi-second-order optimizers, especially their proposed method SOAP, which leverages preconditioning to mitigate gradient conflicts. Extensive experiments across 10 PDE benchmarks and achieving better accuracy and faster convergence.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. The idea of formalizing and quantifying gradient conflicts via alignment scores in PINNs is novel. 
2. The paper provides solid theoretical definitions (alignment score), along with analysis on why quasi-second-order methods mitigate conflicts. The proposed SOAP optimizer is well-motivated and rigorously tested.
3..Writing is very clear.

**Weakness**
1. The theoretical analysis focuses mainly on 1D Laplace equations at initialization.
2. What's the results of Adam+ L-BFGS as it is a very useful technique in training PINNs.
3. The sensitivity of performance to learning rates in SOAP vs. Adam is not deeply discussed.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-16