Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation
Authors: Dmitry Bylinkin
Mikhail Aleksandrov
Savelii Chezhegov
Aleksandr Beznosikov
Venue: ICLR 2026 Poster (2026)
Keywords: Physics-informed neural networks
Multi-task learning
Saddle-point problems
Scientific machine learning
Primary area: optimization
TL;DR: Adaptive weighting of losses corresponding to equations and boundary conditions during PINN training

Abstract: Physics-informed neural networks (PINNs) have gained prominence in recent years and are now effectively used in a number of applications. However, their performance remains unstable due to the complex landscape of the loss function. To address this issue, we reformulate PINN training as a nonconvex-strongly concave saddle-point problem. After establishing the theoretical foundation for this approach, we conduct an extensive experimental study, evaluating its effectiveness across various tasks and architectures. Our results demonstrate that the proposed method outperforms the current state-of-the-art techniques.

Published as a conference paper at ICLR 2026
ENHANCING STABILITY OF PHYSICS-INFORMED NEU- RAL NETWORK TRAINING THROUGH SADDLE-POINT REFORMULATION
Dmitry Bylinkin 1.2
Mikhail Alcksandroy123
Savelii Chezhegov!
Aleksandr Beznosikoy
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) MMoscow Independent Research Institute of Artificial Intelligence 3 Lomonosov Moscow State University
*Sher AI Lab
5 IInooplls University
ABSTRACT
Physics-nffommed neural networks (PINNs) have gained prominence in recent years and are now effectively used in a number of applications. However, their performance remains unstable due to the complex landscape of the loss function. To address this issue, we reformulate PINN training as a nonconvex-strongly con- cave saddle-point problem. After establishing the theoretical foundation for this approach, we conduct an extensive experimental study, evaluating its effectiveness across various tasks and architectures. Our results demonstrate that the proposed method outperforms the current fate-of-the-ar techniques.
INTRODUCTION
Mathematical physics is a cornerstone of modern science. It provides powerful tools for theoretical studies and finds applications in practical fields. One of its central challenges is solving partial differential equations (PDEs) (Bateman 1932 Evans 2022). They arise in the formal description of phenomena ranging from heat diffusion to quantum mechanics and typically take the form of a boundary value problem involving differential operators on some domain (Yakubov and Yakubov 1999,. Generally, there is a system M, equations and M M,,, boundary/initial conditions:
R. (z) f((x), i e [1, M,], x E Ω2
= E [M, + 1, M], x E an,
where f.g. Rd + R are the scalar functions; R. R4[u]. [. : Rd + R are the operators actions the domain set and its boundary,
on the mapping u Rd R.; Ω C IRd and de C gd- respectively. Since exact solutions are rare outside idealized cases, the community is focused on developing numerical methods. Among the most established techniques those based on finite differences (Courant et al.) 1967). volumes (Patankar and Spalding 1983). and elements (Courant al. 1994). Despite high accuracy and computational efficiency of traditional approaches, they require substantial time to interpolate new solution (Grossmann et al. 2024. Figures 4b,6b), (Liu al. 2024b, Figure 6d-f). This limitation makes them impractical in problems where runtime the primary performance metric. A promising direction for addressing this issue lies in machine learning, due to the low inference time of small neural networks (Guo al. 2016 Zhhu and Zabaras 2018 Yu et al. 20188) Although the concept of approximating the solution with parametrized func- tion (0) is quite old and dates back to the works of Meade Jr and Fernandez [994); Dissanayake and Phan-Thien| y1994 Lagaris et al. (1998), it has only recently gained attention under the name neural network) (Raissi et al., 2019). While initial results in this area were learned activations (Jagtap et al. 2020a b), memory (Krishnapriyan et al. 2021 Cho et al. 2023 Liu et al., 2024b) and attention (Zhao et al.
PINN
obtained using MLPs, advanced architectures such
2023 Anagnostopoulos al. 2024) have led significant improvements. PINNs trained via

---

Published as a conference paper at ICLR 2026
the empirical risk minimization (ERM)
(Raissi et al. 2019)
M,
M Σ 1 M,+1
Nr with (0) [R([u(0]([=)]-
min BERd
( = 2 (0) +
(o)(
- f(
Nr
1m1
N₂
Lb. (0) = No
Σ B,[u(0)((a})
-
where [xp))==
{ n= are the sets of samples belonging to the interior and boundary of ?
respectively; N,, Ng are the sizes of corresponding datasets.
While achieving speed up compared to traditional approaches, PINNS face significant challenges during the training phase. There is no guarantee that arg mingeR L(0) minimizes every (,) and (0) individually. dissimilar magnitudes (Hwang and Lim 2024 Figure 2). Consequently, the parameter updates be- come biased toward one of the loss terms, and the solution approximated only on the boundary or only inside the domain (Hwang and Lim 2024 Figure 1). Most successful approaches employ
practice, their corresponding gradients (0), (0) have highly
weights क = . JM
selected from the set S. typically the unit simplex, to balance com-
peting losses for R. R./u]. 8, [u] [Wang et al. 2021 Jin et al. 2021: Wang et al. 2022 Son et al. 2023 Hwang and Lim 2024). The Situation is further complicated by the fact that optimization
landscape substantially varies over physical problems (Krishnapriyan inadeequua al., 2021 Figure 3). There- fore, a scheme that performs well for one PDE may turn out to be inadequate for another (Hao et al. 2023 Table 3), and selecting an appropriate optimizer requires case-by-case search.
In our work, we consider training PINN as a non- non-nonnniinn -on-Euulidean saddle- point problem (SPP):
M.
M
min max []],,,)], with Σ T. (0) OER TES
Σ TyLo.i - AD (T](#),
(2)
j=M.+1
where f-1+f) is the Bregman divergence Nemirovskij and Yudin 1983). Thus, if some operator underestimated relative another one, its weight is increased, as does contribution the loss function. We also introduce the hyperparameter À to enable control over the weights via the penalty for deviating from the reference distribution म. typically the uniform one. similar methodology was considered in (L.iu and Wang (02T). However, the authors provided no theory and examined the Euclidean case, which may be unsuitable if S has a complex geometry. For example, if S is a unit simplex, then KL-divergence is the preferable distance measure, particularly because it accounts for relative rather than absolute changes in weights.
There is currently guaranties for the nonconvex problem ( and this setting remains empirically underexplored for PINNs. In this work, we overcome both thcoretical and practical challenges to investigate the feasibility of training physics-informed neural networks non- non-Euulidean SPPs.
2 RELATED WORKS
2.1 LOss RESCALING IN GENERAL CASE
Training a pwysics--nnneeeeeeee informed neural network is special case of multi-task learning (Zhang and Yang. 2021}. Various rescaling techniques had been developed in this domain by the time of the emergence of PINNs. Chen et al. (2018) suggested treating weights trainable functions Tm (0). They defined separate loss such that the norm of single task gradient V (0)C (0)) is close the sum of the other gradients. A similar approach was explored in (Kendall et al. 2018). However, using neural networks evaluate the parameters leads increased memory consumption. As consequence, the community has developed number of computationally less expensive techniques. [Sener and Koltun (2018) proposed solving a quadratic optimization problem on a unit simplex to determine L.11 [Tm Jm==1 Furthermore, approaches that calculate weights via zero- and first-order statistics have gained attention due to their combination of efficiency and quality (Liu et al. 2019 Yu et al. 2020
Heydari et al. 2019 Chen et al. 2018 Wang al. 2020).
2.2
LOss RESCALING IN PINNS
The unique challenges posed by PDEs and physical constraints motivated the development of weighting techniques specifically for PINNs. Wang et al. (202)) were among the first in this di-
2

---

Published as a conference paper at ICLR 2026
rection. Inspired by ideas behind Adam. (Kingma and Ba 2014). they proposed learning rate annealing procedure that automatically tunes { (Tm L Jm=1 by utilizing the back- prok-propagated gradient statistics. To mitigate the high variance inherent in the stochastic nature of updates, the authors sug- gested computing the actual weights as a running average of their previous values. This scheme was then understood in greater depth (Jin al., [2021] [Maddu al.. 2022] Bischoo and Kraus) (025). As an orthogonal approach, in (Wang et al. 2022). loss rescaling was addressed from a neural tangent kernel perspective. Despite the advances, it may be computationally expensive. Indeed, the use of the Jacobian poses challenge when solving nonlinear equations, as it is not constant in that case (Bonfanti et al., (024). In parallel to these commonly used approaches, a number of exotic non- benchmarked techniques exist. For example. schemes based on likelihood (Xiang et al. 20.2, Hou et al. 2023). augmented Lagrangian (Son et al. 2023) and conjugate cone (Hwang and Lim 2024).
2.3 NONCONVEX-STRONEE
CONCAVE SPPS
The theory of SPPs is constructed mostly for
objectives (Korpelevich 1976 Ne-
mirovski, 2004 Du and Hu 2019 Adolphs et al., 2019 Beznosikov al. 2023). However, the problem 2 falls outside of this class, since the complex nature of differential operators implies poor non-conver landscape 0. On the other hand, in terms of the weights π, [(, π is guar- anteed to be strongly concave regardless of the PDE being solved. Noncomvex-comean (N-C) and nonconvex conconvex-stronggl concave (N-SC) SPPs remain poorly understood. Today's 's research focuses on modifying two-timescale gradient descent-ascent (TT-GDA), which has demonstrated success in training GANs (Heusel et al.] 2017). Using a double-loop scheme, NNuuihhed et al. (2019) achieved an e-solution in O (x*/=² /²²) iterations, where A denotes the condition number of the objective in the concave component. Assuming max-oracle to be available. [Jin et al. (2019) improved this result to (wt (Thekumparampii et al. 2019 Kong and Monteiro 2021). However, nested loops are challenging to implement and tune in practice. At the same time, providing theoretical analysis directly TT-GDA posed a challenge. This was finally done in (Lin et al. 2020) with a complexity of O (**/=²). Later, the result was generalized by Xu et al. (2023). They provided unified analysis of single-loop schemes for N-C problems.
Ô (x²)2). Several triple-loop techniques were developed in
A key drawback of the mentioned methods is the use of Euclidean setting. This may be inappro- priate for describing the geometry of S in the problem 2 (Mohri et al. 2019 Mehta et al. 2024). Consequently, there is an interest in searching for alternatives. Huang et al.
(021) considered
setup that Sun-Eucidean in the non-convex component and Euclidean in the strongly concave
one. However, in our paper, we need the opposite. Indeed, in the problem 2 0 lies in R and is therefore suited to the Euclidean distance, while æ demands a more complicated description. Thus, this work is not suitable for purposes, although it provides useful intuition. [Boroun et al.) (2023) employed F Frank-Woll ploiting non-regularized linear approximation yields sparse values of { (mm) IM which may result in unstable convergence.
(Jaggi 2013) to perform both ascent and descent steps. However, ex-
OUR CONTRIBUTION
Currently, each PDE has its own dominant optimization method: LRA WWag et al.) 2021) for Pois- son, RAR (Lu et al. 2021) for Heat, NTK (Wang et al. [002) for Wave, and Adam (Kingma and Ba 2014) for Navier-Stokes. We study the potential of minimizing the PINN's objective via the non- Euclidean saddle-point problem (2 in order to make the training porcess robust. The paper presents a comprehensive theoretical and empirical analysis of this approach.
Table 1: Comparison of SOTA results. Mean L2RE over 3 runs is used as quality metric.
Approach
Poisson
Heat
Navier-Stokes
Wave
High dim
Prev. best
1.14E-2 8.15F-3
1.74E-2
4.67E-2 2.35F--2
9.20E-2 1.63F--2
4.69E-4
This paper
1.40F-2
1.31F-4
Theoretical foundation. Studying nonconvex-strone concave SPPs with non-Euclidean geome- try of the strongly concave component, we propose method based on a suitable Bregman proximal mapping. We develop rigorous theory, providing guarantees optimization dynamics.
3

---

Published as a conference paper at ICLR 2026
Benchmarking the method. Conducting experiments 22 benchmark PDEs using vanilla PINN, we demonstrate that our approach improves the quality compared to existing optimizers. The pro- posed algorithm achieves SOTA in 77.3% of cases, against 27.3% for the second best.
Extensive empirical study. We show that the proposed weighting scheme reduces the conflict of gradient magnitudes better than its competitors. We attribute this the primary reason for dom- inance of our approach across the majority of PDEs. Additionally, we analyze the computational overhead and examine the robustness of our algorithm changes in hyperparameters.
Challenge-test. We demonstrate the scalability of the proposed method in real-world scenar- ios. Specifically, we train the DoMINO model (Ranade et al. 2025) solve the three-dimensional Navier-Stokes equation on a sample vehicle from the DrivAerML. dataset (Ashton et al. (024).
SETUP
4.1 ASSUMPTIONS
Since our study is motivated by the real-world problem, we address the most general case possible. First, we require the objective to smooth with respect to the Fuclidean norm.
Assumption 1. The function [( π is L-soooth, i.e. for all (01, 11), (02, m2) E IR x S it satisfies
[ m1) (C2, m2) S (0 mT) (02, T2)ll-
Lipschitz continuity of the gradient is commonly imposed in prior work on PINNs (Li et al. 2023
Assumption 1). (Hwang and Lim 2024 Theorem 4.5), (Wu al. 2024 Assumption 3.2). (Liu al. 2024a Theorem 1). While this assumption is generally unrealistic for neural networks (Cybenko 1989). is necessary construct any theory for Deep Learning (Patel and Berahas 2024
To enable more accurate selection of the weights и. we account for the geometry of S by utilizing the Bregman divergence (Nemirovski and Yudin (983).
Definition 1. The Bregman divergence corresponding to the distance generating function up is defined as
Do (mI; n2) = ((m) - xxx22 - (Veb(T2), TI -T))
Earlier, mentioned the example where Di is the Kullback-Leibler divergence. This is particu- larly significant for the purposes of this paper, as we choose S as the unit simplex. However, the theory is established in the general case. Analysis of the problem ( requires D3 to have several basic properties. In particular, Definition is valid only D is bounded from below on S. In the following, present assumption regarding the distance generating function.
Assumption 2. The function vb is I-strongly convex, i.e. for all Tı, T2 E S it satisfies
((TT) 2 ( (T2) + (VD(T))) T2) + 2 n2
x1
Note that this assumption does not reduce the class of neural networks under consideration, as is solely related to the choice of regularizer. Additionally, it holds for all commonly used divergences.
4.2 PROPERTIES OF THE OBJECTIVE
In this section, we obtain several properties of the objective by leveraging its structure.
Lemma 1. Consider the problem (2) under Assumption 2 Then, for every 0 E Rd the function C(O, π) is X-strongly concave, i.e. for all nı, n S it satisfies
L(0, m1) <C(,, π2) + (V C(0, T2). mi x2)
و2) (
See the proof in Appendix E Lemma I shows that the problem 22 is nonconvex-strongly concave
SPP. Hence, it has a single maximum 7 (0) on for every fixed value of
4.3 OPTIMALITY CONDITION
It is challenging to analyze N-SC SPPs using the usual definition of a stationary point. Instead, prior works equivalently reduce it to a stationary point a minimization problem (Huang et al. (022):
((+) L( n" (()).
4

---

Published as a conference paper at ICLR 2026
Since S is bounded convex set. Danskin's theorem implies that & is differentiable with (# Vo (0) [Rockafellar 2015). The common convergence metric employed in the literature is the following (Zhang et al. 2021 2 et al. 2024 Xu et al. 2024).
Definition 2.
(s-stationary point) of (0).. A point 0 an e-stationary point of 2. if
[[)( S 8.
For N-SC SPPs, convergence in the sense of Definition in the standard sense used for SPPs (Lin et 2020, Proposition 4.12).
2 implies convergence to stationary point
ALGORITHMS AND ANALYSIS
5.1 MAIN ALGORITHM
In this section, we follow the trend of investigating N-SC SPPs through modifications of TT-GDA. Adapting it to the problem 2 Due to the complex landscape of the problem to be solved, the algorithmic schemes we rely are extremely simple. Since the parameters may take any value, it suffices to use the classic gradient descent step (Nemirovskij and Yudin
we present Bregman Gradient Descent Ascent.
Algorithm BGDA
Input: Starting point (8", =0) Rd x S
number of iterations T
2: Parameters: Stepsizes 70: 7F. >
1983) to update them (Line 4p. However, the weights are selected from a convex bounded set described by Non-Euclidean geometry. Conse- quently, we utilize the Bregman proximal map- ping (Nemirovskij and Yudin 1983) to perform the ascent step (Line 5). The subproblem in Line requires estimating statistics of the objective only once and therefore does not pose any significant computational difficulties compared to the basic descent step. Moreover, it often has closed-form solution. For example, if Da is the Kl.-divergence (Nemirovskij and Yudin 1983), then
3: for t = 0, ..... T do
4:
g++1 = 0 70 VoL(O' ,
5:
π 1+1 arg min {9(T)} where
((T) = -/m (V=L(0' n T) + (T)
6: end for
exp(1>(V (V, TT));
i= (x (V, (0(CC((TT)))) +T)) im1
In the analysis Algorithm [ it is fundamental to utilize steps of varying sizes. One possible explanation is that the landscape of the objective is much better in the strongly concave component. Consequently, more confident steps can be taken to update the weights. The primary theoretical challenge in the analysis of the method to show the convergence of the iterative scheme based on the metric given in Definition 2 Indeed, for each value of the model parameters 6* there is an optimal point (6*). To address the technical difficulties, we must show that the method generates sequence of points {(Ot, x²) 1== for which the distance between T' and n (8*) decreases when increasing Moreover, we have to account for the non-Euclidean geometry of the problem.
Lemma Consider the problem under Assumptions {(0', x²) x')]T that
Then, Algorithm produces such
DT(TT" x )1
(,
2 (
64x2
where k = 1//x is the condition number of C(A, T) in ग.
See the proof in Appendix Lemma 2 how the distance between the current weight iterate x" and the ideal response Tr" TT"("") evolves over time. This is key result needed to prove convergence. Indeed, since we consider the Euclidean setting in the nonconvex variables 0, the standard inexact
gradient descent analysis implies
T-1 Σ ( O (TOL3) Do(T m²).
((0'+1) - x(8%) < -2(70)
Thus, for a sufficiently small step ya. it is guaranteed to neglect the inaccuracy of finding the maxi- mum at the ascent step. By carefully evaluating (m²"(00 m²) from above and selecting appropri- ate 7e. the convergence is obtained. We formulate this fact as a main theorem.
5

---

Published as a conference paper at ICLR 2026
Theorem 1. Consider the problem
2 uunder Assumptions 7
Then, Algorithm
requires
R LL
D D. (80) x²L
(8)),
iterations
E²
achieve an arbitrary E-solution, where E² Σt V ||2 A - x- is the condition number of [(0, x) in IT.
See the proof Appendix G Note that the derived estimate of T is worse than that obtained in
(Huang et al. 2021) for the Euclidean setting. However, if S is unit simplex interseeted with
cuclidaan ball, it can be significantly improved O (MI/=²) (see Appendix for the detailed discus- sion).
5.2 PRACTICAL VERSION OF BGDA
Since neural networks exhibit a complex loss landscape, it is common practice to run adap- tive versions of algorithms, even when their theoretical guarantees do not account for such modifications. Following this trend, we de- velop an adaptive modification of Algorithm In Algorithm 2 the gradient VoL(O', x²) is smoothed with its previous values as running average (Line[p. In practice, this approach aids in identifying suitable descent direction more quickly. Furthermore, we propose accumulat- ing the gradient history to vary the step sizes (Lines 5 6). This method is effective, as the gradient magnitude indicates the loss smooth- ness locally, which leads to more confident steps and faster convergence. A practice-driven bias correction of calculated statistics also implemented (Lines 7 8 D. To update model parameters and weights, Algorithm] 2 performs the descent-ascent scheme, identical to Algo-
Algorithm 2 AdaativeeGDD (AdaBGDA)
1: Input: Starting point (0", =²) Rd x S,
number of iterations '
2: Parameters: Stepsizes 70. Yr. >
3: for = =0, 0...., do
4:
mg anmb / + (1 - (1) VoL(0' x²)
5:
u = azug + (1 - a2) (Vo((( +
6:
++1 = Bug + (1
x²))²
x
7:
At-1 = 1-a
m
8:
9
#+1 = 1-a
o
9:
H+ = 1-37
0++1 0¹ 70 6
%
11:
T +1 arg min {g(T)), where
q(x) = - ( ( /ott) TT) x²)
12: end for
rithm Namely, Adaptivel iGDA utilizes Adam (Kingma and Ba 2014) and RMSProp (Xu
al. 2021 to perform descent and ascent steps, respectively. See Table 2 for justification.
NUMERICAL. EXPERIMENTS
employ vanilla PINN with 5 hidden layers of size 100. We initialize hyperparameters of Algorithm 2 as 72 = 0.1, 78 = 0.008, af = 0.9, al = 0.999, 30 = 0.999, 0.01. 7o is scheduled linearly with the final value of 0.0004. To tune the baselines from PINNacle (Hao et al.) 2023), we follow the authors' recommendations; for the remaining methods, selected using the same procedure for algorithm. Empirical analysis is conducted with NVIDIA TESLA A100 with 80 GB of GPU memory.
In Sections 6.1 6.4
hyperparameters
In Section 6.5 we work with the DoMINO model developed by NVIDIA (Ranade et al. 2025 To train Adam, we use tuning provided in the code attached to (Contributors) 2023). In experiments with Algorithm 2 we initialize hyperparameters as 7x 0.1, T 0.002, of 0.9, al - 0.999,
go = 0.999, À = 0.1. 7a decreases linearly to 0, 001 during 500 epochs. For this part of empirical study, we use NVIDIA H200 with 140 GB of GPU memory.
6.1 EXPLORING VARIANTS OF ADAPTIVITY
During the empirical study, we used Poisson 2d-C to test various combi- nations of adaptive techniques, such as Adam (Kigmma and Ba| (014) and IMSProp (Xu al. (021). It was Adam+RMSP rop that turned out to be the best one. We attribute this to
Table 2: Comparison of approaches to adaptivity in Algo-
rithm Mean L2RE over 3 runs is used as a quality metric.
Approach
Adam+RM RMS
Adam+Adam
RMS +RMS
L2RE
8.15E-3
4.45B-2
6.02E-1

---

Published as a conference paper at ICLR 2026
the fact that Adam allows account for the poor loss landscape in 0 via gradient smoothing, while the landscape in π is strongly convex, and steps along the current gradient are more appropriate.
6.2 VALIDATION ON PINNacle BENCHMARK
We provide extensive comparison of AdaBGDA (Algorithm 2 with existing approaches. To evaluate the learning potential and generalization capabilities of our approach, we consider 22 partial differential equations sourced from PINNacle (Hao et al. 2023) that covers broad spectrum of PDEs: complex geometry, multiple domains, varying coefficients, long time. As competitors, we consider all methods presented in PINNacle (Hao al. 2023): LBFGS (Byrd al., 1995), Adam (Kingma and Ba 2014), Mu 1t Adam (Yao et (023). and combinations of Adam with RAR (Lu
al. 2021 LRA (Wang al. 2021). NTK (Wang al. 2022).
Table 3: Comparison of AdaBGDA (ours) to the existing techniques. In all experiments, the model is trained to the performance limit. L2RE is used as quality metric. We highlight the best and the second best results for each PDE.
Optimizer
PDE
LSFGS
CRA
NTK
IAdam
AdaRGnA
1d-C
(144420::00-22 (1:33-00000)-22
(1 0200020002)22
(3.00:032)882
(4.96:2-03399-2
(1229:00011):22
Burgers
26-C
(2.7240.32)8-1
0886
2.830.31184
(3.42:0.22))5
(3:26:::44688-1
(4:20::0.1008:
24-C
(3.4120.1515
NaN
(1.11=00918-1
1.14200111822
(7.58:20.62)8:1
(2.79:::.25)8-22
08.15:20.000--3
2d-CG
(5.5020.61)6-2 (2.95=00000-1
0.135=0.22/E22
(6.64:0.50)E-1
(2.76:
(1:70:20.51)1-2
Poisson
CG
(3.945022))-1 1-100-
0.07
(8.73 32)1
(8.58:10.38)EE11
(3.56:10.4438_.1
(6.41:10.21)0:22
2d-MS
(6.61=:0:9)E-1
(1:6::00))E00
0.64
(7900:0.400)-
(6.52:+0.35)E-1
(6.23=+-033//-1
(3443:1..8)EE1
Vd.VC
(2.5810.27)8-11
(2,2810 14)6-1
0:29
0.2
10))
(4.94:1:.56)8..
(2.99:0.1190..
24-MS
(6.71:0060)):2
(1.74:0.10VE:
18.65:1221522
(7:93:10.53)E:2
(2.05:10.18:E-1
(1.40:10.351E22
Heat
(3.83:0:47)1222
18.54
(1.16=0.12)E
(1:20:101100:1
22.58:10.171:22
(7.68:10.691E-2
(249::0.11)0-2
24-LT
(9.8820.0))-1
(1:00:00000-0
(9. 09.97=
0.00::0.00:44
(9.98:00044)5-1
(9.98:4004488-
09:66:2.00))-11
2d-C
(5.6-:0.35)B-2
(2.11=0005)6-1
NaN
2.01
(4.31=20.31):-1
(7.03:9.75)8-11
NS
2d-CG
12.15
NaN
(2 (..)6 03:22-0.2216-1
(3:26:20.21)E-1
(4.31:40.339--1
(7.12:20.270
24-TT
(9.91
10.40.-
(..70-0.87)--1
0.050
(9. 1..
(9.99::0.01)EA)
(1.00::0.00/:1
Id-C
NaN
(9.20=20.82165 03,65-
(5.62::0.57)E-
(1211=:0.10E--1
(,,6::0
Wave
24CGG
(16662002)840 (1:332000F--0
(1.53:00 10y:40
(1008:-0002::4
0..30:1
(1.224000E00 (0366000)-00
0.397
(1003:00.00)000
(1.32:0.081)+0
(1001:0.011:++
0.00EEE
(1.5820000--1
NaN
(9. 2576-2.05)
(2 1605-.
(9: (0:10:0.7440:22
(9.66::0000)8-22
(9.29:20.00):-2
Chaotic
KS
(9.94110.09)E11
NaN
(9.58 0.03)EE
(
(9.69:10.10)E-1
09.51:1
PNd
(2.662000098-3 13:0-
(4.8720 580
<4 (42)-:
(3.37:20.26)8-3
14:08::0.111---
01.11:20.16/::4
High dim
HNd
(3.071.0.00E1
(1.13
0.07
(3 (3.0
(3.71:
(3.00:10.043E-1
(1.35:10.15)E:4
Plar
1330
NaN
0.1)E.
(1.77:10.25)6.1
(9.53:1..5))E:2
(1.32.10.083E.1
66.11:0..23/:22
Inverse
(9223
NaN
(5.08_007))-2
(3.77:1038)))
(1)59:0.11)E80
(7.87:0033388.2
(4.33:10.27)E-2
Table 3 shows that AdaBGDA dominant in 77.3% of cases. In 18% of its rows, we outperform the closest competitor by more than a factor of two; in 36% of its rows, AdaBGDA improves L2RE by more than 30%. At the same time, the previous best optimizer dominated only 27.3% of the benchmark (2.83 times worse than our result), and among those cases, only out of 22 PDEs shows an improvement larger than 5.4% %.
For certain PDEs (Heat 2d-LT, NS 2d-LT, Wave 2d-MS), AdaBGDA yields large L2RE values, sim- ilarly to other optimizers. We attribute this behavior to the fact that the vanilla PINN architecture too simple for such problems. To demonstrate that the saddle-point formulation of PINN training has the potential to consistently improve model quality across a wide range of PDEs even in the case of more expressive architectures, we train sfate-of-the-art model to solve three-dimensional Ntvier-Stokes problem in a domain with complex geometry in Section 6.5
6.3 EXPLORING THE CONFLICTING GRADIENT'S
Table 3 illustrates the stability of the proposed method under changes in problem type, bound- ary/initial conditions, and domain geometry. To numerically investigate this phenomenon, we measure the ratio X tions into groups I = [o. 10000), I2 = [10000, 20000), I = [20000, 30000) and examine the distributions of X1. X2. X3. including their means XT. Xz. Xa and variances 01, σ2 03-
(TC.(00)//lvvc.ee)) while solving Poisson 2d-C. We break the itera-
7

---

Published as a conference paper at ICLR 2026
the dynamics of NTK (Wang et al. 2021). This optimizer is state- of-the-art for the selected PDE. From the first epochs, || (0)] demonstrates significant supe- riority over [0)ll At this stage, we observe X1 2487, 01 = 2352. During the next group of iterations, these ratios hold approximately the same level X2 2342, d2 1628; and af- ter another 10000 they decrease to X 1998, 03 1360. Thus, at the beginning of opti- mization, the value X rapidly concentrates ex- tremely far away from the desired case of equal magnitudes and then slowly decreases. Conse- quently, PINN overfits to the boundary condition.
In Figure I
2000
1000 1500
(b)
(c)
(a) 0 10000 10000 20000 20000 30000 epochs
epochs
epochs
Figure 1: Dynamics of X IV (02// vc.(0) during optimization via NTK. The experiment is made on Poisson 2d-C.
The training process of our method is signifi- cantly more stable. shows results for the proposed iveapiiveBGDA. Using this scheme, obtain X1 7,
Dosiyy
Demitty
Dumulty
X2 25, d2 27:
35
170
200
200
400
X3 = 45, J3 = 127. The pathology is much less pronounced. The resulting improvement is statis- tically significant. Indeed, for / only R 9% of the values obtained with NTK fall within the 3011- interval for Adapti ivBBDDA. At the same time, for /2 and I3 such values do exist all.
(b)
(c)
(a) 0 10000 10000 20000 20000 30000 epochs
epochs
epochs
Figure Dynamics of X = (VC during optimization via AdaBGDA (ours). The experiment is made on Poisson 2d-C.
The superiority of our method is particularly well demonstrated by the error heat maps. Such comparison is presented in F the right part of Figure we observe significant region within the interior of the domain where the approximated solution exhibits large error. The absence of such a region on the left side of Figure illustrates that we successfully address the issue of underestimating losses in the interior of the domain.
Error
Error
0.4
0.4
0.01
0.01
0.2
0.2
0.00
0.0
0.0
0.00
-02
00.11
-0.2
-0.4
0.02
-0.4
-0.01
-0.5
0.0
0.5
00.5
0.0 X
0.5
Figure 3: Heat maps of signed relative errors of PINN trained to solve Poisson 2d-C. AdaptiveBGDD (left) is compared with NTK (right).
6.4 EXPLORING THE COMPUTATIONAL OVERHEAD
One of key characteristics of an optimizer the trade-off between performance and computa- tional overhead. Since AdaBGDA (Algorithm 2 includes additional update to weight competing losses, conducting such a study is particularly important.
Figure shows a direct comparison of the actual runtime of AdaBCDA and its competitors on the Wave Ia-C problem. AdaBCDA achieves convergence approximately 2.5 times faster than of the-art scheme for this PDE. The intersection of deviations at the beginning of training is associated with the rapid convergence of methods. Notably, the model reaches a higher final performance when trained with the proposed algorithm.
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of time/space complexity of AdaBGDA and competing methods on Burg- ers Id-C: (a) Adam, (b) LBFGS, (c) LRA, (d) NTK, (e) RAR, (f) MultiAdami (g) SSBroyden, (h) NNCG, (i) AdaBCDA. The second row of the table shows the time (sec) for
1000 iterations. The third row shows the peak GPU utilization (GB) storing the optimizer states.
Method
(a)
(b)
(c)
(d)
(c)
(f)
(
(h)
Time
8.24
562.07
22.43
19.90
8.33
14.09
176.69
13937.6
7.64
Space
0.23
0.26
0.50
0.47
0.25
0.45
10.66
2.68
0.37
We also provide ablation study of time-per-iteraton and memory consumption (see Table 4 We restrict this experiment to a single run, since all runs were performed on single GPU, and the contribution of noise sources, such variations temperature, is negligible. Additionally, use several methods that are not the part of PINNacle, namely SSBroyden (Kiyani et al. 2025) and NNCG (Rathore al., 2024). Below, we formulate the list of core observations and insights.
AdaBGDA does not experience an increase in iteration time. In- deed, in the case of the unit simplex with KI.-divergence, the ascent Bregman step has a closed-form expression in terms of the values the objective components. Thus, updating the weights requires only forward pass, which is already performed for updating the model parameters. Consequently, AdaBCDA does not incur higher computational cost than first-order methods such as Adam.
GPU utilization also does not increase compared to competing methods. We attribute this the fact that the number of model parameters (40K in our experiments) the number of weights (no more than 11 in PINNacle). Conse- quently, optimizer states for the weights do not inflate memory re- quirements.
156 080
significantly larger than
Figure Comparison Adaptives idaptiveBGDA.o competi- tors on Wave ld-C.
In light of the above, we suggest that saddle-point based approaches have potential to be as compu- tationally efficient as Adam while achieving quality comparable LRA/NTK.
6.5 EXPLORING THE SCALABILITY
Previous sections demonstrate superiority of AdaBGDA over competing methods in terms of L2RE, training fairness, and computational/memoryy efficiency. However, these results are obtained with vanilla PINNs, which are hardly usable in real-world scenarios. Consequently, an important question remains open: is the proposed approach scalable to complex modern architectures?
To address this issue, we consider automotive aerodynamics as a challenge-test problem. We fix a single vehicle geometry from the DrivAerML. dataset (Ashton al., (024) and solve the three- dimensional Navier-Stokes equation for incompressible fluid predict the velocity field and vol- ume/surface pressure. Fluid density and kinematic viscosity are p 1.226 and = 1.507 10-5 respectively. We use the DoMINO model (Ranade al. 2025 38M point-cloud based architec- ture that leverages local geometric information to predict flow fields at discrete points.
Table Comparison of Ada (ours) and competitors on 3d Navier-Stokes. We measure L2RE of: (a) x-velocity, (b) -100-y-wee y-velocity (c) z-velocity, (d) volume pressure, (e) surface pressure.
Metric (a)
(b)
(c)
(d)
(e)
Adam (333940.10)E-1 (8.60-1.73)E--1 (7.16:0.23))--1 (4.55+0.23)E-1 (2.71100.55E-1
LBFGS (3.62:1..3)E--1 (9.56-10.51)E-11 (8.23:4051)EE-1 (4.88:00288EE-1 (3.42.10.1))E-1
AdaBGDA
(2.78:10.14EE-1 (5.99-00.35E--1 (5.34-0.13)E-1 (2.8910.27)E-1 (2.69-00.22)E-1
Table shows that employing the SPP approach leads to fair and robust training process not only in simple settings, but also for large-scale models and complex PDEs. While preserving the quality of surface pressure predictions, AdaRGDA achieves statistically significant improvements

---

Published as a conference paper at ICLR 2026
across the remaining metrics compared to competing methods. Thus, the effect of balance across all components of the objective preserved as model scale and task complexity increase.
6.6
OTHER EXPERIMENTS
Beyond the main empirical study, we also present several additional results. Appendix reports ex- periments with several variants of the vanilla PINN evaluated on PDEs from PINNacle. In Appendix AdaBGDA is compared to competing maddle-point approaches: dual-dimer (Liu and Wang 2021 and AL-P INN (Son et al. 2023). Our scheme dominates in 77.3% and 63.6% of the cases, respectively. Appendix [examinss the robustness of AdaRGDA to hyperparameter tuning. Finally, Appendix D provides comparison between theoretical and empirical convergence of AdaRGDA, demonstrating that numerical observations are consistent with theoretical guarantees.
7
CONCLUSION
In this paper, observe that existing weighting schemes for PINN training do not achieve fully balanced optimization process. To address this issue, we reformulate the training problem as the nonconvex-strongly concave SPP of non-Eaclidean nature. In addition to theoretical analysis, conduct a comprehensive empirical study. We observe a significant increase in the model quality (Table B while preserving the computational efficiency of optimizer (Table We empirically eval- uate an increase in the fairness of the training process (Figures | ( Experiments also demonstrate the scalability of the proposed scheme (Table 5). Thus, the ability of the saddle-point approach to balance conflicting loss terms is crucial for reducing the cost of optimizer selection. As result, fur- ther research in this direction holds promise for developing robust training frameworks for PINNs, improving their applicability to real-world problems.
ACKNOWLEDGMENTS
The work was supported by the Ministry of Economic Development of the Russian Federation (agreement No. 139-15-2025-013, dated June 20, 2025, IGK 000000C313925P4B0002)
REFERENCES
Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi, and Thomas Hofmann. Local saddle point optimization: A curvature exploitation approach. The 22nd International Conference on Arti- ficial Intelligence and Statistics, pages 486-495. PMLR, 2019.
Sokratis Anagnostopoulos, Juan Diego Toscano, Nikolaos Stergiopulos, and George Em Karni-
adakis. Residual-based attention in physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 421:1 16805, 2024.
Neil Ashton, Charles Mockett, Marian Fuchs, Louis Fliessbach, Hendrik Hetmann, Thilo Knacke, Norbert Schonwald, Vangelis Skaperdas, Grigoris Fotiadis, Astrid Walle, et al. Drivaerml: High- fidelity computational fluid dynamics dataset for road-car external aerodynamics. arXiv preprint arXiv:2408. 1969, 2024.
Harry Bateman. Partial differential equations of mathematical physics. Partial Differential Equa- tions of Mathematical Physics, 1932.
Aleksandr Nikolaevich Beznosikov, Alexander Vladimirovich Gasnikov, Karina E Zainullina, A Yu Maslovskii, and Dmitry Arkad' evich Pasechnyuk. A unified analysis of variational inequality methods: variance reduction, sampling, quantization, and coordinate descent. Computational Mathematics and Mathematical Physics, 17-7744 2023.
Rafael Bischof and Michael A Kraus. Multi-objective loss balancing for physics-informet deep learning. Computer Methods in Applied Mechanics and Engineering, 439:1 17914, 2025.
Andrea Bonfanti, Giuseppe Bruno, and Cristina Cipriani. The challenges of the nonlinear regime for physics-iniee informed neural networks. Advances in Neural Information Processing Systems, 37: 41 41852-41881 2024.
10

---

Published as a conference paper at ICLR 2026
Morteza Boroun, Erfan Yazdandoost Hamedani, and Afrooz Jalilzadeh. Projection-free frree methods for solving conconvex-concatt saddle point problems. Advances in Neural Information Processing Systems, 14553844553855 2023.
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu, A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing,
1995.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning. pages 793-803. PMLR, 2018
Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork based meta-learning for low-rank physics-informed neural networks. Advances in Neural Information Processing Systems, 36:11219-1122 2023.
PhysicsNeMo Contributors. Nvidia physicsnemo: An open-source framework for physics-based deep learning in science and engineering, 2023.
Richard Courant, Kurt Friedrichs, and Hans Lewy. On the partial difference equations of mathemat- ical physics. IBM journal of Research and Development, 11(2):215 234, 1967.
Richard Courant et al. Variational methods for the solution of problems of equilibrium and vibra- tions. Lecture notes in pure and applied mathematics, pages 1-1, 1994.
George Cybenko. Approximation by superpositions a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
MWM Gamini Dissanayake and Nhan Phan-Thien Neural-network- approximations for solv- ing partial differential equations. communications in Numerical Methods in Engineering, 10(3): 195-201, 1994.
Simon S Du and Wei Hu. Linear convergence of the primal-dual gradient method for convex- concave saddle point problems without strong convexity. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 196-205. PMLR, 2019.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Society, 2022.
Tamara G Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb. Can physics-informed neural networks beat the finite element method? IMA Journal of Applied Math- ematics, 89(1): 143-)4)444174441 2024.
Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 481 199, 2016.
Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, et al. Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes. arXiv preprint arXiv:2306.08827, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ali Heydari, Craig Thompson, and Asif Mehmood. Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions. arXiv preprint arXiv:/912. 12355, 2019.
Jie Hou, Ying Li, and Shihui Ying. Enhancing pinns for solving pdes via adaptive collocation point movement and adaptive loss weighting. Nonlinear Dynamics, 111((6))1533--1526,,, 2023.
Feihu Huang, Xidong Wu, and Heng Huang. Efficient mirror descent ascent methods for nonsmooth minimax problems, Advances in Neural Information Processing Systems, 34:10431-10443, 2021.

---

Published as a conference paper at ICLR 2026
Youngsik Hwang and Dongyoung Lim. Dual cone gradient descent for training physics-informe neural networks. Advances in Neural Information Processing Systems, 01998633-95599 2024.
Martin Jaggi. Revisiting foofe-wolfe: Projection-free free sparse convex optimization. In International conference on machine learning, pages 427-435. PMLR, 2013.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Locally adaptive activation func- tions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A, 476(2239):202003344 2020a.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Compulational Physics, 404: 109136, 2020b.
Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Minmax optimization: Stable limit points of gradient descent ascent are locally optimal. arXiv preprint 1rXiv:1902.0061 2019.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. In The Twelfth International Conference Learning Representations.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412::1412.69800 2014.
Elham Kiyani, Khemraj Shukla, Jorge F Urbán, Jérôme Darbon, and George Em Karniadakis. Which optimizer works best for phfrree-informed neural networks and kolmogorov-arnole net- works? arXiv preprint arXiv:2501.16371, 2025.
Weiwei Kong and Renato DC Monteiro. An accelerated inexact proximal point method for solving nonconvex-concave min-max problems. SIAM Journal on Optimization, 31(4):2558-2585, 2021.
Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 12:747-756, 1976.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances in neural infor- mation processing systems, 34:26548 00066556666600006-101111160 2021.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 955)9887-1000, 1998.
Ye Li, Song-Can Chen, and Sheng-Jun Huang. Implicit stochastic gradient descent for training physics-informed neural networks. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 37, pages 8692-8700, 2023.
Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-coneave min- imax problems. In International conference on machine learning, pages 6083-6093. PMLR, 2020.
Dehao Liu and Yan Wang. A dual-dimer method for training physics-constrained neural networks with minimax architecture. Neural Networks, 136: 12-125, 2021.
Qiang Liu, Mengyu Chu, and Nils Thuerey. Config: Towards conflict-free training of physics in- formed neural networks. arXiv preprint arXiv:2408. 104, 2024a.
12

---

Published as a conference paper at ICLR 2026
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871-1880, 2019.
Xin-Yang Liu, Min Zhu, Lu Lu, Hao Sun, and Jian-Xun Wang. Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics. Communications Physics, 7(1):31, 2024b.
Haihao Lu, Robert M Freund, and Yurii Nesterov. Relatively smooth convex optimization by first- order methods, and applications. SIAM Journal on Optimization, 28(1):333--3) -344, 2018.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208- 2021.
Suryanarayana Maddu, Dominik Sturm, Christian L. Müller, and Ivo Sbalzarini. Inverse dirichlet weighting enables reliable training of physics informed neural networks. Machine Learning: Science and Technology, 3(1) 30):0150226, 2022.
Andrew J Meade Jr and Alvaro A Fernandez. The numerical solution of linear ordinary differential equations by feedforward neural networks. Mathematical and Computer Modelling, 19(12):1-25, 1994.
Ronak Mehta, Jelena Diakonikolas, and Zaid Harchaoui. Drago: Primal-dual coupled variance reduction for faster distributionally robust optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional conference on machine learning, pages 4615-4625. PMLR, 2019.
Arkadi Nemirovski. Prox-method with rate of convergence (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1):229-2555 2004.
Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
Maher Nouiched, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32, 2019.
Suhas V Patankar and D Brian Spalding. A calculation procedure for heat, mass and momentum transfer in diceessimnii parabolic flows. In Numerical prediction of flow, heat transfer, turbulence and combustion, pages 54-73. Elsevier, 1983.
Vivak Patel and Albert S Berahas. Gradient descent in the absence of global lipschitz continuity of the gradients. SIAM Journal on Mateeaatic of Data Science, 6(3):602-626, 2024.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Rishikesh Ranade, Mohammad Amin Nabian, Kaustubh Tangsali, Alexey Kamenev, Oliver Hen- nigh, Ram Cherukuri, and Sanjay Choudbry. Domino: A decomposable multi-scale iterative neu- ral operator for modeling large scale engineering simulations. arXiv preprint arXiv:2501.13355 2025.
Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training pinns: A loss landscape perspective. In International Conference Machine Learning, pages 42159-42191. PMLR, 2024.
Ralph Tyrell Rockafellar. Convex analysis((pms-28) 2015.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
13

---

Published as a conference paper at ICLR 2026
Hwijae Son, Sung Woong Cho, and Hyung Ju Hwang. Enhanced physics informed neural networks with augmented lagrangian relaxation method (al-pinns). Neurocomputing. 548:126424, 2023.
Kiran K Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Efficient algorithms for smooth minimax optimization. Advances in neural information processing systems, 32, 2019.
Nuozhou Wang, Junyu Zhang, and Shuzhong Zhang. Efficient first order method for saddle point problems with higher order smoothness. SIAM Journal Optimization, 34(4):3342-3370, 2024.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in physics-informed neural networks. SIAM Journal on Scientific Computing. 4335):33055- A3081, 2021.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 10768, 2022.
Zirui Wang. Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874. 2020.
Haixu Wu, Huakun Luo, Yuezhou Ma, Jianmin Wang, and Mingsheng Long. Ropinn: Region op- timized physics-informed neural networks. Advances Neural Information Processing Systems, 37:110494-11052, 2024.
Zixue Xiang, Wei Peng, Xu Liu, and Wen Yao. Self-adaptive loss balanced physics-informed neural networks. Neurocomputing, 496:11-34, 2022.
Dongpo Xu, Shengdong Zhang, Huisheng Zhang, and Danilo P Mandic. Convergence of the rm- sprop deep learning method with penalty for nonconvex optimization. Neural Networks, 139: 17-23, 2021.
Qiushui Xu, Xuan Zhang, Needet Serhat Aybat, and Mert Gürbüzzalaban, A stochastic gda method with backtracking for solving nonconvex (strongly) concave minimax problems. arXiv preprint arXiv:2403.07806, 2024.
Zi Xu, Huiling Zhang, Yang Xu, and Guanghui Lan. A unified single-loop alternating gradient projection algorithm for nonconvex-concave and convex-noncomcave minimax problems. Math- ematical Programminng 201(1):635-706, 2023.
Yakov Yakubov and Sasun Yakubov. Differential- -operator equations: ordinary and partial differ- ential equations, volume 103. CRC Press, 1999.
Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. Multiadam: Parameter-wise scale-invariant optimizer for multiscale training of physics-informed neural net- works. In International Conference on Machine Learning, pages 39702-39721. PMLR, 2023.
Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving varia- tional problems. Communications in Mathematics and Statisties, 6(1):1-12, 2018.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering. 19821, 2022.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33: 5836, 2020.
Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of swweoonvee-sttronly--oocee minimax optimization. In Uncertainty in Artificial Intelligence, pages 482-492. PMLR, 2021.
Yu Zhang and Qiang Yang. A survey on multi-task learning. 1EEE transactions on knowledge and data engineering, 34(12):5586-5609 2021.
14

---

Published as a conference paper at ICLR 2026
Zhiyuan Zhao, Xueying Ding, and B Aditya Prakash. Pinnsformer: A transformer-basee framework for physics-informan neural networks. arXiv preprint arXiv:2307.1183,, 2023.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018
15

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Introduction
2
Related Works
2.1
Loss Rescaling in General Case
2.2
Loss Rescaling in PINNs
2.3
Nonconvex-Strongly Concave SPPs
3
Our Contribution
Setup
4.1
Assumptions
Properties of the Objective
4.3
Optimality Condition
5
Algorithms and Analysis
5.1
Main Algorithm
5.2
Practical Version BGDA
6
Numerical Experiments
6.1
Exploring Variants of Adaptivity
6.2
Validation PINNacle Benchmark
6.3
Exploring the Conflicting Gradients
6.4
Exploring the Computational Overhead
6.5
Exploring the Scalability
6.6
Other Experiments
10
Conclusion
10
Additional Experiments
17
Another SPP Reformulations
17
Robustness to Variations in Hyperparameters
18
D
Comparison of Theoretical and Empirical Results
18
Strong Concavity of the Objective
19
Proof of Lemma
20
Proof of Theorem
22
Enhanced Rates on Regularized Simplex
23
Stochastic Setting
28
16

---

Published as a conference paper at ICLR 2026
To ensure reproducibility, attach the code: hatess//aaoyyrous..eee....ccdodd pinns-bgda- 6
A ADDITIONAL EXPERIMENTS
In this section, we provide additional information to accompany the work. In addition, we use some variations of vanilla PINN validate the theoretical insights.
gPINN. is known that the residual (R, [u] fz) must be zero inside the domain. Consequently, its derivative must also be equal to zero. This approach proposes to modify the objective by adding [R, (t))) 2 as a regularization. In Yuu et al. 2022), it is shown that gPINN has improved quality of the approximation inside the domain $2.
GAAF. This architecture relies on adaptive activation functions (both layer- and neuron-wise).
Jagtap al. 20206) demonstrates the advantages of this approach over vanilla PINNs.
LAAF. Considers GAAF with slope recovery term. For the details, see (Jagtap et al. 2020a).
Table 6: Comparison of AdaBGDA (ours) to the existing techniques. In all experiments, the model is trained to the performance limit. L2RE is used quality metric. We highlight the best results for each pair.
xPINN
LAAF
GAAP
Id-C
0.031=
(..34:0002)E-2
(1:4:=0.02)1:2
(1.28=0002/E--
(5.15=:0.06)1-2
(1.31 0.02/E-
Burgers
2d-C
(3.32:1 0.2//E-
(5.05=0.09))-1
1.0
(4.38:10.07/M-1
(2.97 0.061--11
15.02:10.08)::1
2d-C
(6.83:::0400E--11
0.11::-
(6.10:10.44)E-1
(4.52- 0,31
CG
08.00::0.60)E:1
(4.37
(4.73-10.24)E-
(..16:.0.0))8-2
(8.61
(2.76 0.22)E-2
Poisson
3d-CG
(4.78::0.31)8-1
(3.70=0.35)E-1
(5.67=0.32)E-1
(5.64=0.41)EE2
(8.12:00.8))B-
(9.08-0.57)
2d-MS
(3.3..
(..31110.330.-11
1
(3.74.1..27)--11
(9.1711..1)0--1
(4.14 0.28)E-1
(2.1240.13)E:+0
(1.00::00081E-0
(6:57=0.42/E-1
02.85=:0.1811-11
(8:60:20.1))E-1
07.07=0.43/E-
2d-MS
(1.13::0.07)F-1
(3.94:-0.3//F22
(7.54_0.66)F.2
(1.92:50.12/F22
(9.83-0.71))_-1
(..67=0.35)F.2
Hear
24-CG
(9.47:10..5/E-22
(1.08=0008)E-1
(2.43-10.20)E-22
(4.03:2..34)F-2
(4
(1.23-0.16)F.22
24-LT
(1.00::0.00):+0
-..9:::00111:-1
(9.99=0.01)E-1
09.97:10.011111
(9.99:00001)E-1
0.09--.011:-1
2d-C
0..62::0.62):-22
2..14:0.55)E-22
0.34
(8.21:10.22)E-2
(3.82:10.33)8-22
2.633.02241E.2
NS
2d CG
0.011
0.64).0
(1.26.10.09)E-1
(1.70.10.12)E
(1.05 0.08)E-1
24-LT
0.024=
0.
(9.99=
(9.99=0.01)E-1
00.90=0.01)-1
Id-C
(5.40:10.41)JE.
(2.22.0.12
0.211
(6.84
0.240-2
Wove
(8.11=00061E-1
07.81::0.040-1
(8.23=0.06/E-1
0..911
(7.91:2.05)EE1
07.83=0.031E-1
2d-MS
0
09.11:0..06
0..04=0008)F+0
(1.06:0008)FF
0.07
(2.51::..4)E-11
(..35:0.00)
(9:45:200001-22
(9.55:10.511:-2
0.500:0.05MB-2
(9.30=0.041E2
Chaotic
KS
(9.95-0.04)E-1
0.66:10.03/E-1
0..00=0.02)E+0
-1.00=10.02)E-0
(9.98=0.01)E-1
PNd
3,.00::040)):3
(.26::0.21)
33.13:30.32)E:3
0.621:
(7.83:10.54MB-2
0.231EE3
High dim
HNd
(3.24=0.20)B-1
0.21-=0.82/E-4
(5.13=0.34)B-1
0..31
(5.24:0032))E.
(3.11=0.22)E44
Plav
(8.1240..4)8-2
(..44=0.11E-11
(1.25=0.09)E-1
(9.40:0.711)-2
(2.52=
(1.31=0.121E-1
Inverse
(4.9120441EE00
(6.80=0.52)E-1
(5.53=0.43)E-1
(5.21:0044/E-2
(2.12::0.14BB-1
(6.02=00441)--2
Table 6 demonstrates the dominance of our scheme. The percentage of superiority is 81.8% for
gPINN, 72.7% for LAAF and 90.1% for GAAF. Moreover, there is a significant drawdown only for Burgers 2d-C.
B ANOTHER SPP REFORMULATIONS
In this section, we compare BGDA with approaches based on saddle-point reformulation that have been proposed in the literature. Namely, Augmented Lagrangian relaxation method for PINNs (AL-PINN) (Son et al. 2023) and dual-dime method (Liu and Wang 2021). AL-PINH re- formulates the training of PINNs constrained optimization problem, where initial and boundary conditions are enforced through constraints rather than just penalty terms, and solves a max max-min min problem during training. dual-dimer introduces weights and and additional maximization simi- lar to our methodology, but Ruclidean geometry.
In Table 7 provide comparison of the best achieved L2REs for AL-PINN and dual-dimer with ones provided by our approach. All models are trained to the performance limit. Table 7 demonstrates that our scheme dominates AL-PINN and dua --dd mer in 63.6% and 77.3% of
17

---

Published as a conference paper at ICLR 2026
cases, respectively. The consistent superiority dual-dimer highlights the importance of the non-Euclidean nature of the proposed descent-ascent scheme.
Table 7: Comparison of Adapt i veBGDA the AL-PINN. L2RE is used quality metric. We highlight the best result for each PDE.
Optimizer dusl dimer
AL PINN
AdaDGDA
1d-C
(1.22-10.02)E-22
(1.31:10.02)F-22
Burgers
24-C
(4.54=0007)E-1
(4.21
24-C
(6)044:42)B-1
(4,11=
2d-CG
(3.9810.27)B-1
(7.3420.51)
(1.3::0.11)82
Poisson
3d.CG
(1.5910.12))-11
(4.81
24-MS
(5.0240.3516-
(3.74=0.23)E-1
(347=0.13))-1
24-VC
(2.7::
(6:02=04400B-1
(2.95:0023)8--1
24-MS
(9.4::-07011-33
(8.11=0.63/B-3
(1.87=0.121B-2
Heat
24-CG
(1.12::0.08/F-22
(1.13:20.88)F22
1..7
24-LT
(9.97::0000B-1
(9.95:10.01)E-1
(0:98:10001)E-1
24-C
(100222.00)E-2
(2.33=0,12) 12/E-2
2.21::0.13)B:
MS
2d-CG
(1.11:0.08)B-1
0.466122
(7.70::0.62:8-2
2d-LT
(9.8810002)E-1
(..0EE
(2.7910.22ME.1
10.919
(..6210.09)E-2
Wave
24.CG
(8.02.10.00):11
(..99-0.00))
(..55
24-MS
(1:00:0.031440
(1.00=0.03)E400
0.09-=0.05)
GS
(9.00:10.04)-22
(9.32=0004))-2
(9.31=00.04)B-2
Chaotic
K3
19:62::0.03)b--1
(9.72=0003)E-1
0.052:10.02)8-1
PNd
(8.10::0.11):55
(4.09=0.33)E4
(1.25:00.0))-44
High
HNd
(3.33:
42.60:0.24)E-4
(1.57
(..31:10.12/8-2
(7.4220.11)E22
(7.57-10.15)B22
Inverse
Hlmv
07.22:+0.5018-1
(1.05=082)E40
(4.08=20.1))-2
C ROBUSTNESS TO VARIATIONS HYPERPARAMETERS
In our work, hyperparameters were selected once by tuning to best convergence on Poisson 2d-C from PINNacle (Hao et al. 2023). In this section, study the sensitivity of EdaptiieeBDD choice of hyperparameters. In this experiment, we use Burgers Id-C. Let us start with varying the descent " and ascent "/ step sizes. dembonsttmssssstss robustness to variations in step sizes. This
the
Table 8: Robustness of
to variations in 78. Yr L2RE is used as a quality metric.
70
0.001
0.001
0.001
0.004
0.004
0.004
0.016
0.016
0.016
Ya
0.01
0.1
0.5
0.01
0.1
0.5
0.01
0.1
0.5
L2RE 1.26E-2
1.30B-2
1.28E-2
1.30E-2
1.31E-2
1.31B-2
1.31B-2
1.30E-2
1.35E-2
allows to obtain satisfactory results the benchmark experiments (see Table without additional tuning for each specific PDE. We note that AdaptiveeBCl is also robust to poor tuning of
Table 9: Robustness of Adapti variations in L2RE is used as quality metric.
0.001
0.005
0.01
0.05
L2RE
1.30E-2
1.26E-2
1.26E-2
1.31E-2
D COMPARISON OF THEORETICAL AND EMPIRICAL RESULTS
18

---

Published as a conference paper at ICLR 2026
Convergence guarantees of BGDA established under the Lips- chitz continuity of V.. Despite the idealized nature of this as- sumption, it provides a reasonably good approximation of real loss functions (Khromov and Singh). Figure iillustrates a comparison between the empirical behavior of method and the theoretical convergence. Since theory guarantees a decrease in the gradient norm, we measure the convergence of BGDA according to this cri- terion. Since the convergence bound contains constants that cannot be computed in practice (I, K, ((0)) ( p (09) )7),))))) ),,,, we define the theoretical convergence function [(L) c/ and check whether there exists constant C' such that the plot of f(t) lies within the confidence interval the curve corresponding the gradient norm. In Figure 5 C 20811.
Themeticl
487
Figure 5: Comparison of theory and practice for AdaptivBBGDA
On the logarithmic scale, it can be seen that the empirical curve decreases at the same rate as the the- oretical reference: the slopes of the lines nearly coincide. and the discrepancy between them remains stable throughout all iterations. This confirms that the actual convergence behavior of BGDA aligns with the theoretical predictions, and that the theoretical guarantees adequately reflect its practical dynamics.
We also provide comparison of the convergence speed of Adapt veBGDA against the competing methods on Burgers Id-C. See Figure for the results.
Loss:
AdaptiveBGDA (Our Paper)
NTK
MultiAdam
LRA
10
Loos
10
10-4
250
500
750
1.000
1250
1500
1750
2000
Time
Figure 6: Comparison of Adapt iveBCA to competitors on Wave Id-C. Training MSE loss is used as metric.
E STRONG CONCAVITY OF THE OBJECTIVE
In this section, we prove Lemma I. follows obviously from the form of the objective (see 2 and Assumption 2
Lemma 3. (Lemmal) D.. Consider the problem 2 under Assumption 2
Then, for every O Rd the
function C(0, π) X-strongly concave, i.e. for all Ti, 22 E satisfies
X C(0, m) - C(0,m2) + (V (2) m2) (Du(TI,T2) De(
Proof. Note that (.C. (Z)). The function C(0, is justrongly concave related to Dy, if C(O, 3 - (T) (L et al. 2018 Therefore, the objective is X-stongly relatively concave.
19

---

Published as a conference paper at ICLR 2026
F PROOF OF LEMMA 2
We begin the presentation of the analysis with key result guaranteeing convergence. It demon- strates that the distance between x² and the exact maximum of π 77(00)) has a suitable dynamics with increasing
Lemma 4. (Lemma 2 Consider the problem tuning
2 uuneer Assumptions
2
Then, Algorithm 7 with
A
1
70
412²
184
produces such {(O", +T))E= that
[ (0++),, ",=,,,,x,+))))) x(t))= < (1
De (TT (0'), 2667+8+())))1(000))
64x2
where K = 1//2 is the condition number of C(0, T) in n.
Before proceeding to the proof, let us recall the three-poine identity. It plays key role in the analysis of Bregman methods. (2, y) (x 2) Dv(2, = (V)(=) v 2).
Proof.
(3)
To begin, we use equation in the form
DT(TT" (0++1) n +1) -Do(m (0 n"' (0')) + Do (0'), (0')
(4)
+ ((v((= (0t) V ( n/ (0*+2) x"" (81)).
Further, we write the optimality condition for Line 5
Yr V=C(6', (('(f'==')>>>))))'r + (V +11) V }] x (0') - xt| 2 0.
Applying equation 3 we obtain
-7x (V,L ). ." (B') - + + Do (TT (8)) .|) (TT
( (T x (8'), x1 Du ( ,
After re-arranging the terms, we get
(x( (8'), n + S (TTT x') - De (+1
1+ -²) - 7 ((', n²) )) x (0') - - (5)
Since T²" (00) is the exact maximum of L( л)) in n, there another optimility condition
To (V,L(0', x- (0')), (0') m) 2 0. Substituting x'+1 and summing with equation 5 we derive Dx (0'), ," 1 < (TT (0')
( =T)
+ (V=(("'==*())) V (C((("='),, ( (6') - 15x1-x+11) x111) < <Do(T'(0), D ,T')
7 "V='"0'===''0""" VzL(O' 5T') ,²),"'(0 (09) 7²)
+ 7 C('' x² (0') C T²)) T²), a¹ x11).
Now, we going utilize the strong concavity of [(0, () in m
Im <V=L(0' " (0') - V=L(O' , x (0') - 7²) < Yad
Del (0())(0'),77
2
Thus, we have
(0') < (1- (1-
Tod
2 Des (T (0') - -D (TTtt),
+7 (V=C(0', - V=L(0t, x²), )x rt1)
Next, we apply Cauchy-Schward inequality to the scalar product and obtain
x1 < (
And
(T
D (T( x²) (T 1T²)
2
ar
|V T (0')) v=L(O', '
Ya || m 2112
2
2a
20

---

Published as a conference paper at ICLR 2026
Using L-smoothness of L (see Assumption
we obtain
D=(=" +1) <
1
7ad
D& (T (T) - (TT
2
+
|I= - T'1
1 1| - 11.
2
2a
Since & is 1-strongly convex (see Assumption
2), we have
b |
- - Ds
Thus,
(0') $ (1- 2 D (((T'('') - - x²)
+ (y(((0) x') + Im De x((11).
Choose a = Tr. We can derive
Du(*T((0) x 1
(1-
7xd
+ t22 (x(
2
Since 7n = 1/4L3, we have
(o(=*(0', xx x111) 1
x(1-
1-
1
(o(T)()',
(6)
16k?
Let us return to equation 4 Note that
Va(( (6')) Ve( À (V=L(0' x11)) ==((0' n-" (8)))
Thus, there is
(p( x1 = ( (T (0)+1), ) >" (8')) + DT( ) x(11)
+ LV' ,(11)- V.L(At Tt" x00"), x (0++1) - -= (8*))
(pptT x ((²) + Dos ( (0'), xtt1)
aT?
+ (T (0'), nt ~ + Aa
( (0++1), ( (8")).
Let us choose a = /2/3211. With such a choice, we have
(T (0'+(())),,, +1 S 33xx Dx( (0 n (00))) + 1+ 32k²
(0')
To deal with Do (0'),, ) ग. ,) we utilize equation 6 As a result, we obtain
+ (1
(x( + < 33ts Do (0'+11))) x" (0')
32x² DT n²).
(7)
The rest thing is to prove that the descent step does not dramatically change the distance between the optimal values of weights. Let us write down two optimality conditions:
(V=L(0', T (0')), T (0')) < 0,
(V (( x (o(+1)), π n" -((++1)) S 0.
Let us substitute x** 10'+1) into the first inequality and x x= (0*) into the second one. When summing them up. we have
x (0(++))) x- (0L+1) x (0*) < O. (V.L(0',T T (+0))) V v,
(8)
On the other hand, we can take advantage of the strong concavity of the objective (see Lemma and write
(V L(0' π (0++))) v,C(0', -" (0')), T' (0c+2) x- x00)))
> Dx(T" (0++)))))))) + (TT 2
(9)
x()))
(0)))
21

---

Published as a conference paper at ICLR 2026
Combining equation [ and equation 9
obtain
X² [Du(T" x (0*+))) + ( ( = -
"0'))| -000) -10001-000 - 1²1
Applying the strong convexity of distance generating function (Assumption and re-arranging terms, we obtain
D=(=" ( (0*+2)) + ( ( x (0*)
=4x 8011 - 4 Vo ( +'1
Next, we ass and subtract (0' and apply Assumption
[ We obtain
(Tt (0'),=* x (B(+))) + ( (B(+1), x* (0")) +
L' + (1()1)
Thus, equation transforms into
DT (*( x 1 S 1 (1- 32k²
+ 5287gn9
D (,44+(000,+)++++4444444114000000
With To S I/18EN*1, we obtain
x S (1
1
D=( Ds(m"(00+1) (,
Do
64x²
This completes the proof.
G PROOF OF THEOREM
Theorem 2. (Theorem Consider the problem 2
under Assumptions
7
2
Then, Algorithm 7 with
tuning
A
43
Yn 412
79 S
92 *
33792 A
requires
H:t L.
De (nT (8"),
iterations
to achieve an arbitrary e-solution, where e2 T ET [V A -0())) K = 4/x.
Proof. One can note that 4 is 3mm-ooothh Indeed,
(Q)) - (22))) =I [[V[[((,,=. VoL(02, (02))11
<T. [|00 - 00112 + + (T))),, x (0z)) S I.2 (1 + 4x2) 110 - 0211
<9x2 - 0211
Thus, we can write
((+++1) <0(0") + (V gr+l - 8¹) + 3xL 1012 8011
< - n0/ 3 x1)2
7o (V0(0') VoL(0', =)), (4')))
<6(0")
70
( +3 3 C( n )² + 78 V VoL(O' 7²)112
2
2
<D((")- 70
( + (70 6gHxn)) |VV((0) VoL(O', x')12
Note that
70
62/xI) s
4378
2
92
On the other hand,
(20
10
+ -6821)
S 5700
2
Thus, have
4379
((+++1) ~0(0")
112 70 (o( - VoL(0', m²)
92
4379
<Φ(0')
|V0(0))
)|(
2 12 (T''' (0'())))0))))) (0)
92
22

---

Published as a conference paper at ICLR 2026
Let us denote 8 1 - 1/6ax.. Lemma
transforms into
Do(=" x²) = = (0"" + 26 Σ 25-1-1119(0))
==0
Hence,
4370
((0++1) <0(0")
270 270 (,000)))""+)
92
528
3
6
L
8
Let us sum up over the iterates and obtain
$(0²) <.((")
43
V=(00)11
2% L
Σ
8' (
92
+ 528 3
L
8
1=0
Next, we use the property of geometrie progression and write
43%9
((0") <0)
128 L2 Do (0") 79)
92
T
+ r. 8
L
Choosing 70 S V
92+33792 K Thus, we derive
43
1
1 LAs
K² L2 (m=(m(=)) )0²),=²))
\ s
T t=1
T
T
H ENHANCED RATES ON REGULARIZED SIMPLEX
The theory presented in Appendices G is constructed for and arbitrary Bregman divergence. This the main reason for the deterioration of the theoretical guarantees compared to the Euclidean setting. In this section, we look towards the selection of the efficient approach for determining the set of weights S. We consider classic approach of using a unit simplex 4 M-1
M
A M (TI...., TM)
Am O,
1,
112=1
EM= Ty log Tm goes to infinity at vertices of AM-1. Thus, one cannot guarantee smoothness of C(O, π) in for every fixed 6, To avoid this, we propose to intersect the simplex by euclidean ball. This approach common in the literature (Mehta al.. 2024). Thus, we deal with
Note that ('( -M
S AM-1 BB-n R).
where 1t = T/
Lemma 5. The function L(A, σ) 4--smooth म, i.e. for all Ti T2 E S satisfies [V TT) VL(0, T2)l - L || -Tol2.
Moreover, under strong regularization (R < 1), it is
L R)
23

---

Published as a conference paper at ICLR 2026
Proof. Without loss of generality, consider (a, .., where min Tm. Note that
[2 = A diag
TM
Thus, we need to find max
EE A M-1 with || - 1112 - RR. Let us write
- 411
M (M
(M- - 1)
R².
(10)
M
Consider b 1-a
Then equation 10
transforms into
(1-aM)² R². M 1)
M
Solving the one-dimensional optimization problem, find the Lipschitz constant of Vx, x). If R < 1, then
A
AM
La 1/M O(R) 1 MO(R) s *OAM2 nt).
Note that this value is negligible. Indeed, E (0, and M in problems of mathematical physics (see equation [ is usually equal to 3-4. Thus, if Ki L=/X appears in the estimate, it is comparable in magnitude to other constants hidden in the big-O.
Now let us move to an analysis with enhanced rate.
Lemma 6. Consider the problem under Assumptions
Let S AM (1, R). Then,
Algorithm 7 with tuming
Ya
4 ,
70
184x3 RL
produces such {(0", (T=) that
DT (1'+))))) n (1
1
1
D (20))))))))222242111111111
64
where K= L/>, Kxx LT/2.
Proof. To begin, equation 3 in the form (T (0L+1), x (+1) = (m (0 ( + Do (0') (0'), x(11
(11)
(V+((T ( (0'+1) x"" x0())).
Further, we write the optimality condition for Line 5
Yar V V,(" n²) + (v Vv(((')) (8') - xlt1) 2 0.
Applying equation B we obtain
-Yx ((' nt) T T²), (0') - + Du (0'), - D (m (0') x211) - ( mm² 2 0.
After re-arranging the terms, we get
(n (0'), m (T( (0"), n²) D ( m²) - 7x C(0' m²), x (0') x( (12)
Since TT" (8t) is the exact maximum of L(0 π.) in ग, there is another optimility condition
7x (V=L(0', x (0') )), TT" (0*) m) 2 0.
Substituting x = x1+1 and summing it with equation 12 we derive
DTp f0(T=((0)) (0)),=--) < (T (T')
(T +1 x²)
+ (V, x
V ;-),, n' (09) - x x(11)
< (0')
1,
+ (V.((0), x =(0")) V=C(0', ,,,)=²))))) ,a'),=²(0' - x²)
+ (V,L( x =0())) V.C(0' -²),, n² n x1).
24

---

Published as a conference paper at ICLR 2026
Now, we going to utilize the strong concavity of [(8, TT) in x:
I'm <V,C(0', x (0*) - V==(00,= 'T" (0') - x²) s -1x1
2
Thus, have
n 11 < (1- 7od
2 (T= (0'),=²) D ( x²
+7x (Vx T (00)) V C(0' x²)x, x5 - x(t1)
),
Next, we apply
inequality to the scalar product and obtain
D=(=" ) S (1 (1- 7nd
D (T (0'), x²) (m ++1 m²²
2
+ at
I( x (0')) V=L(0', x')11 + Ya || - 1
2
2a
Using 4n -mooothness of [(0, π) in x (see Lemma
5) we obtain
Do (T (0'), ) π 1 = 7xd
D Pu(=T((') (')) - (u( ,x0
2
+ 7x aL
|| |+* 111 + 7't 1| -- 12
2
2a
Since 1-strongly convex (see Assumption
2) we have
2 n -
- - (u(TI,
Thus,
(0'), +11) S (1- Ind 11-2z)
D D=( - -
2
+ a o Du (8)) ) T²) + Ya
Do x
a
Choose a = Yn. We can derive
D= (0)), S <11-
Tad
+
2/2
Do (0'), (0)),
2
Since 7/ = a/a23, we have
De( x x11 s
1
1
Do(="((')))))
(13)
16x
Let us return to equation 11 Note that
1 ī (V=L(0' ; ,111- V V=L(O', -- (0')))
Vi((T (0')) V 41
Thus, there is
Dy(T ( ) = D vo (e(((==((t)))) (BL+1), T"' (09)) + D2s (B')) a(11)
+ V L(0'' x x11) - V=L(0', TT' =T(0'),; x (00+)) - - --=-"(00) (0'))
<D p) ( (0) T" (0') DT (0'), x111
aL D ( ( (0"), n ++ +
1 Du 7,="),;²000))
Aa
Let us choose a = 42/s215. With such a choice, we have
1 Do(
(T( ( ), x1+1)== S 33kg Dx (0'+1), n (²())) + 1+
32k²
To deal with DT(TT ,1) ), we utilize equation
13
As a result, we obtain
DT( 10 42) - 33m4 Das (x ((0()))))).)))) (0')) (1
1
(a((""(0',, (14)
32k?
25

---

Published as a conference paper at ICLR 2026
The rest thing is to prove that the descent step does not dramatically change the distance between the optimal values of weights. Let us write down two optimality conditions:
(V,L(0', x - (6')) 0,
(V, ग x T"" -
- 0.
Let us substitute (0++)) into the first inequality and म (0*) into the second one. When summing them up, we have
(VxC(0", ' =0"))) V c(0't) x =+* (0++1) (0")) S O.
(15)
On the other hand, we can take advantage of the strong concavity of the objective (see Lemma and write
(V π (6)+1)) Vx "- ()")), (B++1) x²' x0()))
A (T (0") T- (() + (x (,++),
(16)
Combining equation 15a and equation 16 we obtain
X²
(T "=" (0' + Du (T
( (0*+)), x (0'))) & 1, 11 - x (00))1 11001110000
- 0-12.
4
Here we can not apply the smoothness in æ. Instead, we have to use the smoothness in (0, .). Next, applying the strong convexity of distance generating function (Assumption and re- arranging terms, we obtain
(x( (0"),
(0"), T" ()*+)) (xT x (8") =4x0||0*+1 0¹11² V
VoL(0' T-1
Next, we ass and subtract (0') and apply Assumption (TT a (0(+1)) + (T (0++1), (0'))
[ We obtain
Thus, equation 14 transforms into
(T (0 ) < (1 (1-
+ 528 2 D 2666 (1
32x2
With 70 S 1/18SNKKL we obtain
DT( (0t+)) ), x(+1) < (1
De +200)),))+++00
64
This completes the proof.
Next, we modify the main proof to obtain enhanced convergence.
Theorem 3. Consider the problem 2 under Assumptions
1
2
Let S S
S=
AM-
n B R).
Then, Algorithm with tuning
X
43
7o S
4L2
92 33792 K². xL
requires
KLA + (T (69), T
iterations
E²
to achieve an arbitrary e-solution, arbiirarye-sollution,, where e2 + -T-1 L./X.
7 [ A - x 1/2.
Ki
Proof. One can note that Φ is 3s.L-mmoohh. Indeed,
v4(0,) - 2 T VeL(O,, (02))1 (02))11
s1.² [[|0] 0211 + ( x (0z))] < I22 (1 4x3))101 - 0211
< 12
26

---

Published as a conference paper at ICLR 2026
Thus, can write & 1(0++1) (0') + (VD(0), 0+ -0) + 3x - 0912
<0(0") - moll 3 ,
70 (V - VoC(0',= VO)())
<0(0') - 70 |0000) + Vo x )² + 70 |V - VoL(0' 79)1
2
2
<0(0") ( (2 kr |0000)) + (20 61HnI) ||V((0)) - VoC(O', x')??
2
Note that
10 6.gnT.) S 92
4370
On the other hand,
(
10 2
S 700-
Thus, we have
43
((0++1) <0(00)
(0(00)) 7ol vollV - VoL(O' v
92
43
<0(0")
VV((')) )12 - D (7)) x0))
92
Let us denote&
1 1/6xx.. Lemma transforms into
1-1
Dx (O'))(0'), x²) S 8 d'' +0
2
Σ |
j=0
Hence,
43%0
((++)) <0(00)
2, 2 (+)
92
+ 528
8 [
j=0
Let us sum up over the iterates & and obtain
T-1
T-1
x(0)) <T)
4370
(0'))1) + 27
Σ
(TT (00))
92
T-I6-1
+ 528
Kig
2 1
ΣV |(
Next, we use the property of geometric progression and write
2(0T) <0) - 4370
92
3 2 L
Choosing 70 <
43 K
Thus, we derive
V
92+33792
1
T-1
R.LLAD
L2 (T (8"), x9)
[+ oo o
T
T
T
Above we discussed that K2 is small, since not many equations appear in the PDEs systems. Thus, we can focus on k only and proceed to
(ALAO
Dz ((0))
50
T
T
This finishes the proof.
27

---

Published as a conference paper at ICLR 2026
I STOCHASTIC SETTING
In the current realities of machine learning, it is almost never possible to use all the data to compute a gradient. Motivated by this fact, we develop a stochastic theory for our scheme. Note that the computation V,C(O, т does not need to perform backward. Therefore, we analyze the stochasticity in only. Consider a stochastic gradient Go(0", ,²,E) calculated from one randomly selected sample
Assumption 3. Stochastic oracle Gg is unbiased and light-tailed, i.e. Rg [Go(O, ()) VoL(O, x), E [[Go (0, VoL(0, x)]l S o², v(0, x E E S.
In our analysis, we rely on batching. Namely, we sample subset of data points and use it to approximate the gradient. The main difference between Algorithm B and deterministic BGDA the
Algorithm S-BCDA
1: Input: Starting point (0), )T) Rd x S, number of iterations T
2: Parameters: Stepsizes ye: Yn
do 4: Draw collection of i.i.d. data points {
3: for f. 0, ...., T
5:
0++1 0- 1o B Σ Go n², {!) // Optimizer updates parameters
6:
π _++ arg msn {-7x <V* T)),T) + DU x²)} // Optimizer updates weights
7: end for
8: Output: (ØT, x²)
of stochastic oracle call in Line
Lemma 7. Consider the problem under Assumptions 7
3
Then, Algorithm 3
with taning
X
41,2
70
184K4T
produces such {(0', ; ) that
1
(T( 1 s 64x2
De (0'))
B
where K = 1//x the condition number of C(0, T)) n.
Proof. To begin, equation 3 in the form
(T (0++1), x x111) = =D (0'+1
(0') + D2 (0'), x x(11
(17)
(V
(0++1)) x0))))
Further, we write the optimality condition for Line 6
Yar V C((" n²) + (V Vv(((')) (8') - xlt1) 2 0.
Applying equation B we obtain
-Yx ((, nt)
T T²), (0') - + Do (0'), D (mT (0'), x111) - (MT m 2 0.
After re-arranging the terms, we get
(n (0'), m (T( (0"), n²) ( m²) - 7x (V C(0' m²), x (0') -(+1) (18)
Since TT" (8t) is the exact maximum of L(0 π.) in ग, there is another optimility condition
7x (V=L(0', x )), nT (0') क) 2 0.
Substituting x = x1+1 and summing it with equation 18 we derive
DTp (0(m*(0')) (0') 1-1 < Du (T (TT)) D& ++1 x²)
+ (V, x
v -²), n. (09) - x x(11)
< (
+ (V.(( x =(00) V=C(0' "x','=² (0') - x²)
+ (V,L(0', x =0())) V.C( -²),, ) n x1).
28

---

Published as a conference paper at ICLR 2026
Now, we going to utilize the strong concavity of [(8, TT) in x:
I'm <V,C(0', x (0*) - V V==(0',=' 'T" (0') - x²) s -1x1
2
Thus, have
n 11 < (1- 7od
2 (T= (0'), D ( x²
+7x <Vx 7 (00)) V C(0' x²), )) x(t1)
Next, we apply
inequality to the scalar product and obtain
(0' + S (1 (1- 7nd
( (0'), x²) (T ++1 m²²
2
+ ax
V= x') + Y || - 1
2
2a
Using L-smoothness of L (see Assumption [.. we obtain
< (1 - Aad
Dx( n +2
D&s (TT (0')))),=)) Do x²
+ T,22
|| 10') T'12
Ya -
2
2xx
Since & is 1-strongly convex (see Assumption
2). we have
b T 11-ma T2l1 S (T))
Thus,
(0', S (1 1 1-22nd)
2ord
2 (T (0'),=) (='tttt, + Yx
+ 7rL? 2 D's (TT (
(0'), (0)))==²))))))
a
Choose a = Yr. We can derive
(8") ) T x1+1)= 5
1
7
+ t Dx (
2
Since 7x = 1/aL3, we have
1
(0' (0), s
1
(+())))
(19)
16x²
Let us return to equation 17 Note that
1 Vei( (0))) V x (V+L(0' x11)) - - x80)))
Thus, there is
DelTT" (8*+)) ), = =Dx( ((0+=)))))))))))0))))))) ), (('))
(80)""('),
+ 1 (v. L(0', LiO ,111- - v, n r((()), x (0t+1) - x x(()))
D ( ) x (8') + Dos (T (0'), xt11)
al? ( (0'), x² x ))+ + Aa
1 ) x,
Let us choose a = X2/3214, With such a choice, we have
1
Do (T (0++1), ) < 33x Dx (0 (0'))
1
x00),
32k²
To deal with (0')) n5 +2), we utilize equation 19 As result, we obtain
(e(TT(('+)) (11) < 33x² ( ( x (+')) + (1 32x²
1
DO(T
(20)
29

---

Published as a conference paper at ICLR 2026
The rest thing is to prove that the descent step does not dramatically change the distance between the optimal values of weights. Let us write down two optimality conditions:
(V,L(0', x = (6")) 0,
(V, ग x T"" -
-
Let us substitute (0++) into the first inequality and म (0*) into the second one. When summing them up, we have
(VxC(0", =0'))) V c(0't) x" a+* (0++1) (0")) I O.
(21)
On the other hand, we can take advantage of the strong concavity of the objective (see Lemma and write
(V π (B Vx "- ()))), (B(+1) x²' x+'))
A (Tt (0') T (B + ( x(())
(22)
Combining equation and equation
we obtain
X²
4 [D (0'),, x" (0 + D n (0++)), x (0') })] - 1.21 (0 - -00)) - 091/2.
arraaanniin the terms and substituting Line 5 we derive [T+ x (0)+1)) + (T (0++1 (00)) <4x2110t+1 - -011
B
<476x²
-
Σ
Go (0' n 24)
B
1=1
After adding and subtracting VeL(O', T²) we have
(0tt)), <4 | x0| + 47/n²
VoL(0',
1
Σ
Go & 5t)
B
Let us take an expectation and derive
EDx(T" ) n (0')) <E8 (400)) VoL(0 - -1000))1 47221202
B
<E890 [ 16 | (T (TT)), 473R202
Thus, equation 20 transforms into
EDp(x* (0 +2 <E
528'22012 (
32k²
B
With To 3 1/18AN" L, we obtain
26422xx ( 132
ED (n" (6'+1), (1) s E
1
De( (0(,"
64n²
This completes the proof.
Now let us proceed to the convergence proof for Algorithm 3
Theorem 4, Consider the problem
2 under Assumptions
7
Then, Algorithm 7
with tuning
A
43
xx
7x-
412
70
92
33792
L
B max
g2
requires
K LA + x2 L2 Do (T (0"), 70) + ///202
iterations
E²
achieve arbitrary e-solution, where e² 1 E [V A -(00)) K 1/2.
30

---

Published as a conference paper at ICLR 2026
Proof. One can note that & is 3m.othh Indeed,
[+)())) - (0(02)) =IVVo<(01, T- (0:) - [02(())
< [10] - - 2t (02)) - 12 (1+4xx) - -821 19101=1101 - 02l1².
Thus, we can write
o 1 =0(00) + (V"(0"),0' - 0²) + +xx111104t1 - -0'12
B
1 B
2
=2(0")- - 60 (o(0'),
Gold', (5) ) +333gL
B
B 1=1
1
=.(0'- 70 (0(0)) 70 VQ(("), v2(00)
Gol0', n' S''00'T',$. (1
B
:=1
1
+ [ +')12+ + 673mL
VoL(0' x²)
Go オ|' 5t)
B
Consider an expectation. We have
E0(( (B 70 VQ(())) (VT(0'), VO(0") - VoC(0t,
+ +6122L V= +6)
<BD(0') -
70
2 1278xT.) || [
+ 2("0 + 1272 (0) - VoL(0', x1) + 0rgxL0²
B
Note that
10
- 12. S
4378
2
92
On the other hand,
(2) + 12,gmL) s %00-
Thus, we have
43
E <BD(0')
V4((')) + 70 V - VoL(O' (0)(0','))1=---1 + 673xL02
92
43710
V0(0')) + 200L' (T" (0'),,)))) + 6r3xIT02
<BD(0')
92
B
Let us denote 8- = 1 - 1/64x2. Lemma transforms into
<ES' D& (TT + +6472 6 t-1
ED& (T')), x²)
81 V
+ 1-1
x22
11
B
j=0
Hence,
4370 (V
(00+))50(01))
27 1.2'
D' 70)
92
1-1
6r2
+ 52 8 2
81111 +
8
B
1=0
1-1
26472n L202
Σ
81-1-j
B
1=0
31

---

Published as a conference paper at ICLR 2026
Let us sum up over the iterates t and obtain
T-1
$(0²) < - 43
+ 2
Σ
8' Det ( (T")),
92
t=1
1=1
T 6 bgzxLod B
528 10 1?
8
|
t=1
1=
T-161
3 26 K
12202
81-1-j
B
Next, we use the property of geometric progression and write
%
6(0") <Q((0)
|
128
L
92
6T
168967
:2 T
+ 337927 x L
|~000)12
B
B
Since 7/0 S L. estimate this
4379
T-1
*(0") <00
128 2, (TT x,7²)
92
T-1
ToT62
92 T
+ -37927 nr
A 1
|0((0))1
+
BR
B
Choosing 79 5
92+33792 KT we derive
43
LA (TT
0²
920²
T
T
T
BR
B
Let us choose B T/2 and obtain
1
L
R² L (T
).T))
k 12200
T
Σ
(V
+
T
T
T
t=1
This finishes the proof.
Note that the reasoning could be done for the special case of regularized simplex. Then would obtain improved rates.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
Language models were used to improve text quality (mostly to correct grammatical errors). LLMs were not used obtain theoretical results write code.
32


Reviews:
[{"review_id": "Qw6GqwcOF7", "reviewer": "Reviewer_Zu1m", "summary": "This paper proposes to reformulate the PINN loss as a min–max (saddle-point) problem with a Bregman regularizer to stabilize the weights. Using a simple optimizer, Bregman Gradient Descent–Ascent (BGDA), the authors empirically show improved performance on numerous PDE benchmarks. To explain where the gains come from, the paper presents a gradient conflict analyses showing reduced conflict, which supports the reported performance improvements.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. High novelty. The paper reformulates PINN into a new optimization framework that mitigates gradient-conflict issues.\n\n2. Clear and rigorous. The problem setup and method are explained clearly, and the paper provides rich theory, such as complexity bounds, that demonstrates the strength of the approach.\n\n3. Strong experiments. The evaluation across 22 PDEs is comprehensive and shows the stability of the proposed method.", "weaknesses": "Although this paper have already compared to many baseline optimizers. It would better to include some SOTA method like [1] and [2], especially [2] also experimented on Burgers equation and achieves a much lower error.\n\n\n[1] Rathore, Pratik, et al. \"Challenges in training pinns: A loss landscape perspective.\" arXiv preprint arXiv:2402.01868 (2024).\n\n[2] Kiyani, Elham, et al. \"Which Optimizer Works Best for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks?.\" arXiv preprint arXiv:2501.16371 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZBygORpMTz", "reviewer": "Reviewer_Me5S", "summary": "The paper tackles the instability problem of PINNs by reformulating the training process as a saddle-point problem (SPP). Specifically, the authors introduce trainable loss weights $\\pi$ and impose a divergence regularization term to ensure strong concavity with respect to $\\pi$. This transformation is claimed to improve regularity and stability during optimization. The resulting formulation is optimized using a BGDA algorithm and its adaptive variant (AdaptiveBGDA). Theoretical guarantees are provided for convergence in nonconvex–strongly concave settings, and experimental results on multiple PDE benchmarks demonstrate faster convergence and improved accuracy compared to baseline PINN training schemes.", "questions": "1. How sensitive is the algorithm to the choise of  hyperparameters? \n\n2. Please include computational overhead of the proposed method.\n\n3. Pleae include experimental comparison with dual dimer method\n\n4. How does the method perform on high-dimensional or more challenging PDEs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **clear motivation**: The work addresses a well-known bottleneck in PINN training: gradient imbalance between boundary and residual terms. The motivation for stabilizing the training through adaptive loss weighting is well articulated and relevant.\n\n2. **Novel regularization for strong concavity**: Introducing a divergence-based regularization (via Bregman divergence) ensures strong concavity in the inner maximization problem, which theoretically stabilizes the optimization dynamics. \n\n3. **simple algorithm**: The BGDA framework is conceptually simple and can be implemented with minor modifications to existing PINN optimizers.", "weaknesses": "1. **Overall poor presentation including unlear definitions**: for example, in Equation (1), quantities such as $M_r$, $M$, and the role of $S$ and $\\hat pi$ are not clearly defined. \n\n2. **Potentially incorrect criticism of prior work**: I think the claim that Liu & Wnag (2021) used $\\pi \\in \\mathbb R^d$ and suffered instability due to the unconstrained weight space may be inaccurate. The dimension of $\\pi$ in Liu & Wang (2021) is also the number of losses. \n\n3. **Justification for reformulation**: Transforming a nonconvex minimization problem into a nonconvex–strongly concave SPP can make optimization harder, as it introduces an inner maximization step that must be approximated in practice. The paper lacks a quantitative discussion on whether the stability gain outweighs the computational burden..\n\n4. **Limited theoretical scope**: The analysis guarantees stationarity of the envelope function $\\Phi(\\theta)=L(\\theta, \\pi^*(\\theta))$ but does not ensure convergence to a joint stationary point. Therefore, the derived resutls are somewhat weak compared to standard convergence analysis in the literature on minimax optimization . \n\n5. **Lack of fair computational comparison**: computational costs are not reported. Since SPP training increases per iteration cost, it should be reported. \n\n6. **Insufficient experimental details and reproducibility**: Standard deviations are missing. Deatils for selection for hyperparameters are not rerported. \n\n7. **Missing baseline**: Comparison with dual-dimer approaches (Liu & Wang (2021)) is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpOtMiFqGh", "reviewer": "Reviewer_akXc", "summary": "Although PINNs have attracted significant attention and found wide applications, their performance often suffers from instability due to the complexity of their loss functions. To address this, the research team reformulated PINN training as a nonconvex–strongly concave saddle-point problem. They first established a solid theoretical foundation by studying this class of saddle-point problems under non-Euclidean geometry and proposing a Bregman proximal mapping–based method with rigorous guarantees on optimization dynamics. Extensive empirical validation followed, demonstrating that the proposed method consistently outperforms existing approaches and achieves state-of-the-art results across nearly all benchmark PDEs, effectively unifying previously task-specific dominant methods. Finally, the paper presents a comprehensive theoretical and empirical analysis of the proposed approach and highlights its key contributions: (1) establishing the theoretical foundation, and (2) conducting extensive empirical verification.", "questions": "1. What is the computational complexity (in both time and space) of AdaptiveBGDA compared to Adam, NTK, or AL-PINN? Please provide either a theoretical complexity analysis or detailed runtime profiling (e.g., per iteration cost, GPU utilization, or convergence rate per epoch). This would clarify whether AdaptiveBGDA’s superior stability comes at a computational trade-off.\n\n2. The paper mentions fixed hyperparameters (γπ = 0.1, γθ = 0.008, α1 = 0.9, α2 = 0.999, β = 0.999). How sensitive is the method’s performance to these choices? Please include a sensitivity analysis or guidelines for hyperparameter selection would strengthen the practical usability of the method.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  Reformulating PINN training as a nonconvex–strongly concave saddle-point problem and introducing the AdaptiveBGDA optimizer based on Bregman divergence is innovative. The method offers a fresh perspective distinct from traditional gradient-based techniques.\n\n2. Theoretical analyses are rigorous, with proven convergence guarantees. Experimental results across multiple PDE benchmarks show consistent improvements over state-of-the-art methods.\n\n3. The paper is well-structured, with clear explanations of the algorithms and mathematical derivations, intuitive figures, and transparent proofs.\n\n4. To some extent, it addresses the instability issues in PINN training and has potential application value for solving PDEs in scientific computing.", "weaknesses": "1. The theoretical analysis (Theorem 1 and Lemma 2) establishes convergence under idealized assumptions, but these are not explicitly connected to observed empirical behaviors. The discussion lacks explanation of how theoretical stability guarantees translate to improved convergence in practice.\n\n→ Improvement: Add a section explicitly mapping theoretical assumptions (e.g., strong concavity, bounded divergence) to empirical observations in experiments.\n\n2. Although the paper includes a link to anonymous code for review, it does not ensure full reproducibility—some implementation details and hyperparameter settings are missing.\n\n→ Improvement: Provide complete training scripts, configuration files, and datasets used for each benchmark.\n\n3. The theoretical role of Bregman divergence is mathematically well defined, but its intuitive contribution to training stability in non-Euclidean spaces is underexplained.\n\n→ Improvement: Include visual or conceptual explanations illustrating how Bregman divergence shapes the optimization trajectory or balances competing losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mpajqfx3gd", "reviewer": "Reviewer_xUeL", "summary": "This paper addresses a fundamental challenge in training Physics-Informed Neural Networks (PINNs): the instability and poor performance caused by conflicting gradients from the multiple loss terms (e.g., PDE residuals and boundary conditions). The authors propose a novel solution by reformulating PINN training as a nonconvex-strongly concave saddle-point problem (SPP). The authors provide a solid theoretical foundation for their proposed Bregman Gradient Descent Ascent (BGDA) algorithm, proving its convergence to a stationary point even in this non-Euclidean geometry setting. Extensive experiments on the comprehensive PINNacle benchmark demonstrate that AdaptiveBGDA significantly outperforms a wide range of state-of-the-art optimizers and weighting schemes across different PDE problems.", "questions": "1. See weakness above.\n\n2. Could you give insights on how practitioners should go about selecting the regularization parameter λ and the initial learning rates for a new problem?\n\n3. The experiments on high dimension is simply on d=5 dimensions, which is very weak indeed. Could you report results on higher dimensions like d=100?\n\n4. The theory and experiments consider a small number of loss terms (M). How would the method scale and perform if M were very large, for instance, in problems with a very large number of boundary condition constraints or coupled multi-physics systems?\n\n5. Assumption 1 needs the Lipschitz continuous for the loss function, which is widely used in the optimization literature. However, since the PINN loss is highly non-convex and ill-conditioned, does assumption 1 really hold for the PINN loss? How about the convergence", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The saddle-point reformulation is a principled and theoretically grounded approach to the known problem of loss imbalance in PINNs..\n\n2. The results are extensive, covering wide variety of PDEs (Poisson, Heat, Navier-Stokes, etc.) and challenging features (complex geometry, multiple domains, long time intervals). \n\n3. The proposed AdaptiveBGDA algorithm is a concrete and practical contribution that can be readily adopted by other researchers and practitioners to improve the stability and performance of their PINN models.", "weaknesses": "1. Although rich experiments are conducted based on PINNacle, the relative error values in Table 3 are relatively bad and did not reflect the state of the art results, and the improvement seems modest. For example, the Poisson and Heat equations are simple and can be easily solved by traditional numerical methods with high accuracy, however, the errors stays around $\\mathcal{O}(10^{-2})$, which are too large. For the complex chaotic KS equation, the improvements are from 9.57E-1 to 9.53E-1, however, for the same solution, work [a] in the year 2023 has reported a much better result 1.61E-1.\n[a] AN EXPERT’S GUIDE TO TRAINING PHYSICS-INFORMED NEURAL NETWORKS, 2023.\n\n2. The experiments in the main text seem to use a standard PINN architecture.  However, vanilla PINN architectures are hard to present the sota results[b]. While Appendix A shows results on more advanced architectures, a deeper discussion on how the saddle-point optimization interacts with specific architectural choices (e.g., Fourier feature networks, transformers) would be valuable.\n[b] Causality-enhanced Discreted Physics-informed Neural Networks for Predicting\nEvolutionary Equations,2024\n\n3. While the paper shows a runtime speedup, the saddle-point formulation inherently introduces additional computational overhead per iteration due to the proximal step for updating π\n\n4. The method introduces new hyperparameters (λ for the regularizer, step sizes γ_θ and γ_π). Although the paper uses fixed values across all benchmarks to show robustness, optimal performance in new, highly complex problems might still require tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)