RoPINN: Region Optimized Physics-Informed Neural Networks
Authors: Haixu Wu
Huakun Luo
Yuezhou Ma
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2024 poster (2024)
Keywords: Physics-informed Neural Networks
PINN Training
Deep Learning
Primary area: machine_learning_for_physical_sciences
TL;DR: This paper proposes and theoretically studies a new training paradigm as region optimization. RoPINN is derived from the theory as a practical training algorithm, which can consistently benefit diverse PINN backbones on extensive PDEs.

Abstract: Physics-informed neural networks (PINNs) have been widely applied to solve partial differential equations (PDEs) by enforcing outputs and gradients of deep models to satisfy target equations. Due to the limitation of numerical computation, PINNs are conventionally optimized on finite selected points. However, since PDEs are usually defined on continuous domains, solely optimizing models on scattered points may be insufficient to obtain an accurate solution for the whole domain. To mitigate this inherent deficiency of the default scatter-point optimization, this paper proposes and theoretically studies a new training paradigm as region optimization. Concretely, we propose to extend the optimization process of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from this new paradigm, which is implemented by a straightforward but effective Monte Carlo sampling method. By calibrating the sampling process into trust regions, RoPINN finely balances optimization and generalization error. Experimentally, RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation. Code is available at this repository: https://github.com/thuml/RoPINN.

RoPINN: Region Optimized Physics-Informed Neural Networks
Haixu Wu, Huakun Luo, Yuezhou Ma, Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, China
fuuhx23, www.uuuiii.mmyy200maaaaaaaaaaaaa .ts.s/22000mmmisssssinnaaaaaaa edu. en, {j htta
Abstract
neural networks (PINNs) have been widely applied solve partial differential equations (PDEs) by enforcing outputs and gradients of deep models satisfy target equations. Due the limitation of numerical computation, PINNs are conventionally optimized on finite selected points. However, since PDEs are usually defined on continuous domains, solely optimizing models on scattered points may be insufficient to obtain an accurate solution for the whole domain. To mitigate this inherent deficiency of the default scatter-point optimization, this paper proposes and theoretically studies a new training paradigm as region optimization. Concretely, we propose to extend the optimization process PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from this new paradigm, which implemented by a straightforward but effective Monte Carlo sampling method. By calibrating the sampling process into trust regions, RoPINN finely balances optimization and generalization error. Experimentally, RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation. Code is available at this repository: https: //t:://gitttttbt www.ooo////aoa/ Sonn/thum//RoPIig
Introduction
Solving partial differential equations (PDEs) is the key problem in extensive areas, covering both Due to the inherent complexity of PDEs, they Thus, series of numerical methods have been widely
engineering and scientific research [38
40
49
usually cannot be solved analytically 10
40 or finite element methods 5 7 However, these numerical methods usually suffer from huge computational costs and only obtain approximate solution on discretized meshes [26] 43 Given the impressive nonlinear modeling capability of deep models 4 [4], they have also been applied to solve PDEs, where physics-informed neural networks (PINNs) are proposed and have emerged as promising and effective surrogate tool for numerical methods 44 36 35). By formalizing PDE constraints (i.e. equations, initial and boundary conditions) as objective functions, the outputs and gradients of PINNs will be optimized to satisfy certain PDE during training [36]. which successfully instantiates the PDE solution as a deep model.
explored, such as spectral methods
22
Although deep models have been proven to enjoy the universal approximation capability, the actual 24 35
optimization process of PINNs still faces thorny challenges [
As basic topic of PINNs, Previous methods
the optimization problem has been widely explored from various aspects / 44]
attempt to mitigate this problem by using novel architectures enhance model capacity [58] 3 50),
reweighting multiple loss functions for more balanced convergence [47), resampling data to improve important areas [51] developing new optimizers tackle the rough loss landscape [55 37 etc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Typical
Resampling
Highorder
Variational
Ours
Loss
Loss
Loss
Loworder Loss
Loss
[a] Point Optimization
High-order Loss
High-rrder Teats
(b) Region Optimization
Figure I: Comparison between previous methods and ours. Previous point optimization methods
train PINNs via the loss on selected points, which is different from our region optimization paradigm.
Orthogonal to the is the objective function of PINNs. We notice that, due to the limitation of numerical calculation, is almost impossible to optimize the loss function in the complete continuous domain. Thus, the conventional PINN loss is only defined series of selected points [36] (Figure []]. However, the scatter-point loss function obviously mismatches the PDE-solving objective, which is approximating the solution on continuous domain. This mismatch may fundamentally limit the performance of PINNs. Several prior works also try to improve the canonical PINN loss function, which can be roughly categorized into the following two paradigms. One paradigm enhances the optimization by adding high-order derivatives of PDEs as a regularization term to the loss function [55]. However, calculating high-order gradients is numerically unstable and consumnnn,,, even with automatic differentiation in well-established deep learning frameworks 2 34 The other paradigm attempts to bypass the high-order derivative calculation in the PINN loss function with variational formula- tions [88 19 20]. Nevertheless, these variational methods still face difficulties in calculating the computations, thereby mainly limited very shallow models or relying on massive sampled quadrature points and elaborative test functions
methods, this paper focuses on a foundational problem, which
integral of deep models and will bring
57
This paper proposes and studies in Figure [ we extend the optimization process from selected scatter points into their neighborhood regions, which can theoretically decrease the generalization eπтоr the whole domain, especially for hidden high-order constraints of PDEs. In practice, we seamlessly transform this paradigm into practical training algorithm, named Region Optimized PINN (RoPINN), which is implemented through simple but effective Monte Carlo sampling. In addition, to control the estimation error, adaptively adjust the sampling region size according to the gradient variance among successive training iterations, which can constrain the sampling-bas based optimization into a neighborhood with low-variance loss gradients, namely frust region. In experiments, RoPINN demonstrates consistent and sharp improvement for diverse PINN backbones on extensive PDEs (19 different tasks) without any extra gradient calculation. Our contributions are summarized follows:
training paradigm for PINNs region optimization. As shown
To mitigate the inherent deficiency of conventional PINN optimization, propose the region optimization paradigm, which extends the scatter- poatter-point optimization to neighborhood regions that theoretically benefits both generalization and high-order constraints satisfaction.
We present RoPINN for PINN training based on Monte Carlo sampling, which can effectively accomplish the region optimization. trust region calibration strategy proposed to reduce the gradient estimation error caused by sampling for more trustworthy optimization.
RoPINN can consistently improve the performance of various PINN backbones (i.e. canoni- cal and transformer-basee on a wide range of PDEs without extra gradient calculation.
Preliminaries
PDE with equation constraints, initial (ICs) and boundary conditions (BCs) can be formalized as F())(x) 0, Ω. [z))(x)
Ωn: B(u) (x: 0, z aΩ
(1)
where F, B denote the PDE equations, ICs and BCs respectively 6 R - R is the target PDE solution. x C IR++1 represents the input coordinate, which is usually a composition of
spatial and temporal positions, namely (I1, ... Ld, t). Po corresponds to the t 0 situation.

---

Correspondingly, the PINN loss function (poinr optimization) is typically defined as follows [7
36
No An [F(o))
No
Naw
Ano Noo
[(ua) No
[[((u) Σ
+
(2)
Non
where ue represents the neural network parameterized by 0. No, Not Nan are the numbers of sampled points in Ω2 So, On respectively. X. is the corresponding loss weight. Note that there is an additional data loss term in Eq. (2) when
access the ground truth of some points [36]. Since we mainly focus on PDE constraints throughout this paper, we omit the data loss term in the above formalization, which still maintained in our experiments. In this paper, we try to improve PINN solving by defining a new surrogate loss in place of the canonical definition of PINN loss in Eq. 2 In contrast, the relevant literature mainly improves the objective function in two different directions as follows. Appendix F provides a more comprehensive discussion on other relative topics.
High-order regularization The first direction is to add the high-order constraints of PDEs as regularization terms to the loss function [55]. Specifically, since PDEs are sets of identical relations, suppose that the solution is K-order differential function, Eq. { can naturally derive a branch of high-order equations, where the k-th derivative for the j-th dimension is a
0x. ax. F()(()) = 0,
a EEB E I & (d+ 1).1 k < K, corresponding to the following regularization:
Nk.,
Akii
Ck (u))
F (uo) (2x)
(3)
NR.j
where Ne.. denotes the number of sampled points with weight AR.j- Although this design can explicitly enhance the model performance in satisfying high-order constraints, the calculation of high- order derivatives can be extremely time-consuming and unstable (39]. Thus, in practice, the previous methods [35] 31] only consider a small value of K. In the next sections, we will prove that RoPINN naturally incorporate high-order constraints. Besides, presented in Eq. this paradigm still optimizes PiNNs on scattered points, while this paper extends optimization to neighborhood regions.
Variational formulation As a classical tool in traditional PDE solvers, the variational formulation widely used to reduce the smoothness requirements of the approximate solution [42] Concretely, the target PDEs are multiplied with set of predefined test functions (U1, ... , and then the PDE
equation term the loss function is transformed as follows [18 19 20
M Coqu( = Σ
un(x))
dx, uu (x) dee (4)
0(2,)
where F(a,) defines the antiderivative of F. on the j-th dimension. Using integrals by parts, the derivative operation in F is transferred to test functions { thereby able to bypass high-order derivatives. However, the integral on Q is still hard compute, which requires massive quadrature points for approximation [18). Besides, test function selection requires extra manual effort and will bring M times computation costs (57] In contrast, RoPINN does not require test functions and will not bring extra gradient calculations. Also, RoPINN employs trust region calibration strategy to limit the optimization in low-variance regions, which can control the estimation error of sampling.
3 Method
As aforementioned,, we propose the region optimization paradigm to extend the optimization from scatter points to series of corresponding neighborhood regions. This section will first present the region optimization and its theoretical benefits in both reducing generalization error and satisfying high-order PDE constraints. Then, implement RoPINN in simple but effective sampling-base based way, along with a trust region calibration strategy to control the sampling estimation error.
3.1 Region Optimization
For clarity, we record the point optimization loss defined in Eq. at x as L(ug. x). where x E Ω DoU0 denotes the point selected from inner domain, initial state or boundaries. We adopt S to denote the finite set of selected points. Then Eq. 2 can be simplified as L(uo,S) Exes L(uo, a:).
3

---

Correspondingly, we define the objective function of our region optimization innovatively as
Lr
Cregion ug. S)
Cregion
(ug, x) [2 X S EJ C(ug, x E)de,
1
(5)
S
EES
where Ω2 [0. (d+1) represents the extended neighborhood region with hyperparameter r. Al- though this definition seems to require more sampling points than point optimization, we can develop an efficient algorithm to implement it without adding sampling points (see next section). Besides, this formalization also provides us with a convenient theoretical analysis framework. Next, we will discuss the theoretical properties of the two optimization paradigms. All proofs are in Appendix
Generalization bound Here we discuss the generalization error in expectation pendent of the point selection, therehy quantifying the enror of PINN optimization more rigorously. Definition 3.1. The generalization error in expectation of model trained on dataset S is defined as
[3] which inde-
Egen = Es [F (W)(S), S2) L (uA(s),
where A denotes the training algorithm and A(S) represents the optimized model parameters. Assumption 3.2. The loss function L is I.-Lipschitz and B-smooth with respect to model parameters, which means that V/z E St the following inequalities hold:
[[ue x) [(uo,, x)1 - L||0] -22ll. VVVo[uu, x) Vol(us,: s B||0] - -02ll-
(7)
Theorem 3.3 (Point optimization). Suppose that the loss function L is 1-Lipschitz---smood for O. If we run stochastic gradient descent with step size ar at the t-th step for T iterations, we have that:
(1) If C is convex for and at S
3. then Egen S
21² ΣT= ar (proved by [I3 52 [ST
(2) If C is bounded by constant C for all 0, x and is non-cowvex for 0 with monotonically non-
C 2T-(T-1) (righter bound than 13 521).
increasing step sizes 04 S "i then Egen S [S]
B[||-1]
Lemma 3.4. If C is bounded for all 0, x and is convex, L-Lipschitz---smoon with respect to model parameters 6, then Lregion is also bounded for all 0, and convex, L-Lipsschtz---smoone for f.
Theorem 3.5 (Region optimization). Suppose that the point optimization loss function L is L- Lipschitz and 8-smooth for 9. If we run stochastie gradient descent with step size 04 for T iterations based on region optimization loss Cregion in Eq. (5) the generalization error in expectation satisfies:
(1) If C is convex for 0 and at S 3. then Egen <(1 Del 21 ΣT Cz.
)
(2) If C is bounded by constant C for all 0, x and is non-convex for 8 with monotonically non- 21=
increasing step sizes a4 S 3.. then Egen I [s| A[[S|-1) - JL( [s] where a finite number
that depends on the training property at the several beginning iterations.
Proof. Based on the Lipschitz assumption, Egen can be bounded by term relating to the expectation of distance between parameter 0 optimized from different training sets. The region optimization
paradigm brings a more "consistent" gradient optimization direction than point optimization each iteration, thereby benefiting the generalization property. See Appendix A.3 for complete proof. O
we can observe that region optimization can reduce the generalization error Egen. Furthermore, the region optimization theorem also provides a more general theoretical framework. For example, the conventional point optimization is equivalent to the case of Ω,, = 0, where only one single point is selected for each region. For another extreme case, enlarging the region size to the whole domain (i.e. Ω, = Ω2), Eq. (5) is equivalent to directly optimizing the loss defined on Q, where the generalization eтror will be reduced zero. Unfortunately, this ideal situation cannot be satisfied in practice, since Fq. (5) requires precise calculation of the integral over the whole domain. More discussions of practical implementation are deferred to the next section.
From Theorems 3.3 and 3.5
High-order PDE constraints In our proposed region optimization (Eq. (5)) the integral operation on the input domain can also relax the smoothness requirements of the loss function L. For example, without any additional assumption of the smoothness of L(ug, x)) on œ, we can directly derive the generalization error for the first-order loss 8 Cregion (ng, x) on the j-th dimension as follows.

---

Algorithm Region Optimized PINN (RoPINN)
Input: number of iterations T, number of past iterations To retained to estimate the trust region, default region size r, trust region calibration value do 1, and initial PINN parameters On- Output: optimized PINN parameters 0r.
Initialize empty buffer record gradients g.
for t 0 to 'I" do
// Region Optimization with Monte Carlo Approximation
Sample points from neighborhood regions: S' {ay + -:))_n 2, E S, ~ U[0, =]((+1)
Calculate loss function L. - L (ua,,S S')
Update O2 02+1 with optimizer (Adam 21 L-BFGS 27 etc) to minimize loss function Lt II Trust Region Calibration
Record the gradient parameters 9r throughout optimization
Update gradient buffer g by adding and keeping the latest To elements
Trust region calibration with 0++1 = (g)ll) end for
Corollary 3.6 (Region optimization for first-order constraints). Suppose that L is bounded by C' for all 0, x and is L-Lipschitz and B-smooth for 9. If we run stochastic gradient method based on oirst-order j-th dimension loss function ăz; -
2 Cregion for T iterations, the generalization error in
Theorem 3.5 2) still holds when we adopt the monotonically non-increasing step size on S 2pr
Corollary 3.6 implies that the integral the input domain in region optimization can help training PINNs with high-order constraints, which is valuable for high-order PDEs, such as wave equations. In contrast, this valuable property cannot be achieved by the classic point optimization. See Example
Example 3.7 (Point optimization fails in optimizing with first-order constraints). Under the same assumption with Corollury 3.6 we cannol obtain the Lipschitz and smoothness property of a [lus, x). For example, suppose that [(ug, x) _8 V x E [0, 1](++1,, which I Lippschitz- Dug
1-smooth. However, Vo Be, f(ua, x) unbounded when æ - 0, thereby not Lipschitz constant.
3.2 Practical Algorithm
Derived from our theoretical insights of region optimization, we implement RoPINN as practical training algorithm. As elaborated in Algorithm|] [ RoPINN involves the following two iterative steps: Monte Carlo approximation and trust region calibration, where the former can efficiently approximate the optimization objective and the latter can effectively control the estimation error. Next, we will discuss the details and convergence properties of RoPINN. All proofs can be found Appendix B
Monte Carlo approximation Note that the region integral in Eq. (5) cannot be directly calculated, so we adopt straightforward implementation based the Monte Carlo approximation. Concretely, to approximate the gradient descent on the region loss Cregion we uniformly sample one point within the region Ω2 for the gradient descent at each iteration, whose expectation equal to the gradient descent of the original region optimization in Eq. (5):
Eg- VoL(uu, x + E) VoL (ug, x).
(8)
In addition to efficiently approximating region optimization without adding sampling points, proposed sampling- based strategy is also equivalent to a high-order loss function, especially for the first-order term, which is essential in practice (55). Concretely, with Taylor expansion, we have that:
Eg~U(D)) (V VoL(uu, x + 4)) = (a,) (VaL(us, Φ) + Vo (uo, x)) + O(IEE))))
(9)
where Ω, = [0, r and L1 represents the first order of loss function, namely a C(up, a.).
Theorem 3.8 (Convergence rate). Suppose that there exists a constant H. s.t. Yo and Va: Ω. VeLreioon (us, x)ol S H|| If the step size at - V decreuses over time for iterations,
the region optimization based on Monte Carlo approximation will converge at the speed of
Vol Cregion (ue, x>]l²| 0
1
(TT
(10)
5

---

Theorem 3,9 (Gradient estimation error). The estimation error of gradient descent between Monte Carlo approximation and the original region optimization satisfies:
(gg-)(())))
VoL x + + VoL (uo,
(ug, a)) ])]
()))) (VoL(ug, E)] (11)
where a represents the standard deviation of gradients in region S2,.
Trust region calibration Although the expectation of the Monte Carlo sampling is equal to region optimization as shown in Eq. (8) this design will also bring estimation error in practice (Theorem 3.9 A large estimation error will cause unstable training and further affect convergence. To ensure reliable gradient descent, we propose to control the sampling region size r towards a trustworthy value, namely trust region calibration. Unlike the notion in optimization [56), here trust region is used to define the area of input domain where the variance of loss gradients for different points is relatively small. Formally, adjust the region size in inverse proportion to gradient variance:
r X
(12)
(C-Ur(()))) (Vo a + E))
In practice, we initialize the trust region size as a default value r and calculate the gradient estimation error during the training process for calibration (Algorithm[]) []. However, the calculation of the stan- dard deviation of gradients usually requires multiple samples, which will bring times of computation overload. In pursuit of a practical algorithm, we propose to adopt the gradient variance among several successive iterations as an approximation. Similar ideas are widely used in deep learning optimizers, such as Adam [21) and AdaGrad [48). which adopt multi-iteration statistics the momentum of gradient descent. The approximation process is guaranteed by the following theoretical results.
Lemma 3.10 (Trust region one-iteration approximation). Suppose that loss function L is L- Lipschitz and B-smooth for and the 1-th step parameter is Oc. Two gradient difference sequences be- tween successive iterations, Vo 21 21)- - Voo(uu, z2)l and V V af 21)-VoL(ua, z2)ll share the same limit, as the difference of the two sequences is dominated by the following inequality:
V 21) VeL(ue, 1.22) [V =1) - VeL(ue, =2) S BLaa-1 (13)
where 0t-1 represents the step size at the (1 - 1)-th iteration, which approaches 0 as tends to co. Theorem 3.11 (Trust region multi-iteration approximation). Suppose that loss function L is L-Lipschitz and B-smooth for and the learning rate a4 S Br. converges to zero over time then the estimation error can be approximated by the variance of optimization gradients in multiple successive iterations. Given hyperparaneter To, our mulli-iteraatinn approximation is guaranteed by
lim o (LVoL(uo...,, - 2.)) t-00
= (IVoL(uo,,
(14)
It is worth noting that, as presented in Algorithm 1 since the gradient of each iteration has already been on the shelf, our design will not bring any extra gradient or backpropagation calculation in comparison with point optimization. Besides, our algorithm is not limited certain optimizer, and in general, we can effectively obtain the gradients of parameters by retrieving the computation graph.
we observe that larger region size will benefit the generalization error, while Theorem 3.9 demonstrates that too large region size will also cause unstable training because it will result in excessive gradient estimation error of Monte Carlo sampling in our implementation. The above analysis reveals the underlying trade-off between generalization and optimization of PINN models, which is formally stated below.
Balance between generalization and optimization Recall that in Theorem 3.5
Theorem 3.12 (Region Optimization with gradient estimation error). Based on the same assump- tions in Theorem 3.3 but optimizing the PINN model with the approximate region optimization loss Capprox (ua, a) V Llua, x +), ~ (Q,,) for T iterations, we further denote the upper bound of gradient estimation error Er.grad maxesT [ Co
Lapprox
V VoLregion ||, then Egen satisfies:
(1) L is convex for 9 and a4
2 B. Eyen < (
(1- 11-1111/191/4
Er.grad
22
ET an [8]
inversely proportional to [92,]]
generally [a,1)
(2) If C is bounded by a constant step sizes 04 S 1
C. aand is non-convex for 0 with monotonically non-increasing
C (12(T-1) 11/112111111) + J'Engraa(1 + |
pe. then Eyen
S
|S| 9([S--1)
inversely proportional [0,)]
generally x [0,]
where J' is finite number that depends the training property at the several beginning iterations.

---

Proof. In contrast to Theorem 3.5
here the gradient estimation error will bring extra optimization discrepancy between different training sets. See Appendix for complete proof.
Based on Theorem 3.12 we have pinpointed that classical point optimization (2,- - 0) and sampling points globally (92 = S2) discussed in Theorem 3.5 correspond to two extreme cases. The former makes grad but cannot reduce the generalization error, while the latter holds a large gradient estimation error. Thus, neither case can yield perfect generalization for PINNs. In contrast, the design for calibrating trust regions in RoPINN provides an adaptive strategy to better balance generalization and optimization, which can adjust the region size according to multi-iteration training stability.
4 Experiments
To verify the effectiveness and generalizability of our proposed RoPINN, we experiment with a wide range of PDEs, covering diverse physics processes and series of advanced PINN models.
Benchmarks For a comprehensive evaluation, we experiment with four benchmarks: ID-Reaction, ID- Wave, Convection and PINNacle [12.. The first three benchmarks are widely ac-
Table 1: Summary of benchmarks. Dimension means the
input space and Derivative is the highest derivative order.
Benchmark
Dimension
Derivative
Property
ID-Reaction
ID+Time 1 (e.g- 2s)
Failure modes
knowledged in investigating the opti-
ID-Wave
ID+Time
2 (e.g- 2yy)
mization property of PINNs [7 37
Convection
ID+Time 1 (e.g- du
Failure modes
Especially, ID-Reaction and Convec- tion are highly challenging and have been used to demonstrate "PINNs failure modes" 24 33 As for PINNacle [2]. it a comprehen-
PINNacle 12
ID-5D+Time 1-2 (egg Dx"
16 different tasks
sive family of 20 tasks, including diverse PDEs, e.g. Burgers, Poisson, Heat, Navier-Stoker Wave and Gray-Scott equations in ID to 5D space and on complex geometries. In this paper, to avoid meaningless comparisons, we remove the tasks that all the methods fail and leave 16 tasks.
Base models To verify the generalizabilityy of RoPINN among different PINN models, we experi- ment with five base models, including canonical PINN 36) activation function enhanced models: QRes 3 and FLS [50]. Transformer- -based model PINNsFormer [58] and advanced
backbone KAN 28 PINNsFormer
[58 and KAN
28 are the most advanced PINN models.
Baselines As stated before, this paper mainly focuses the objective function of PINNs. Thus, only include the gradient-enhanced method gPINN (55) and variational-based method vPINN [18] as baselines. Notably, there are diverse training strategies for PINNs focusing on other aspects than objective function, such as sampling- -based RAR [51] or neural tangent kernel (NTK) approaches 47 We also experimented with them and demonstrated that they contribute orthogonally to ROPINN.
Implementations In RoPINN (Algorithm D we select the multi-iteration hyperparameter To from (5, 10} and set the initial region size r = 1o for all datasets, where the trust region size will be adaptively adjusted to fit the PDE property during training. For ID-Reaction, ID-Wave and
Convection, we follow (58) and train the model with L-BFGS optimizer (27) for 1,000 iterations. As for PINNacle, we strictly follow their official configuration
122 and train the model with Adam (21 for 20,000 iterations. Besides, for simplicity and fair comparison, we set the weights of PINN loss as equal, that is X. 1 in Eq. (2). Canonical loss formalized in Eq. (2), relative LI error (rMAE) and relative L2 error (rMSE) are recorded. All experiments are implemented in PyTorch (34) and trained on single NVIDIA A 100 GPU. See Appendix C for more implementation details.
4.1 Main Results
Results As shown in Table 2 we investigate the effectiveness of RoPINN on diverse tasks and base models and compare it with two well-acknowledged PINN objectives. Here are two key observations.
RoPINN can consistently boost performance on all benchmarks, justifying its generality on PDEs and base models. Notably, since the PDEs under evaluation are quite diverse, especially for PINNacle (Table is extremely challenging to obtain such a consistent improvement. We can find that the previous high-order regularization and variational-
methods could yield negative effects in many cases. For example, gPINN [55) performs badly on ID- Wave, which may be due to second-order derivatives in the wave equation. Besides, vPINN [88 also fails in ID ID- Reaction and QRes.
7

---

Table 2: Comparison between RoPINN and other objective functions (gPINN
55 and vPINN (8]
under different base models. Metrics for PINNacle [12] are the proportions of improved tasks over 16 tasks, where full results can be found in Appendix E A lower loss, rMAE or rMSE indicates better
performance. For clarity, we highlight the value with blue if surpasses the vanilla PINN and the best is in bold. Promotion refers to the relative promotion of RoPINN over the vanilla version.
ID-Reaction
1D-Wave
Convection
PINNacle (16 tasks)
Base Model Objective
Loss
rMAE rMSE
Loss
rMAE
rMSE
Loss
rMAE rMSE
rMAE
rMSR
Vanilla
2.0e-1
0,982 0,981 1.9e22 0.326 0.335 1.6e-2 0.778 0.840
gPINN
2.0e-1
0,978
0.978 2.8e-2 0.399 0.399 3.1e-2 0.890
0.935
18.8%
18.8%
PINN 36
<PINN
2.3e-1
0.985
0.982
7.3e-3
0.162
0.173
1.1e-2
0.663
0.743
25.0%
25.0%
RoPINN
4.7e-5
0.056
0.095
1.5e-3
0.063
0.064
1.0e-2
0.635
0.720
93.8%
100.0%
Promotion
99%
94%
90%
92%
80%
80%
25%
18%
14%
Vanilla
2.0c-1
0.979
0.977
9.8c-2
0.523
0.515
4,2e-2
0.925
0.959
gPINN
2.1c-2
0.984
0.984
1.3c-1
0.785
0.781
1.6e-1
1.111
1.222
12.5%
12.5%
QRes 3
VPINN
2.2e-2
0.999
1.000
1.0e-1
0.709
0.721
5.5e-2
0.941
0.966
12.5%
12.5%
RoPINN
9.0e-6
0.007
0.013
1.7e-2
0.309
0.321
1.2e-2
0.819
0,870
81.3% 9%
81.3% %
Promotion
99%
99%
99%
83%
41%
38%
71%
11%
Vanilla
2.0c-1
0.984
0.985
3.6e-3
0.102
0.119
1.2c-2
0.674
0.771
gPINN
2
0.978
0.979
9.2e-2
0.500
0.489
3.8e-1
0.913
0,949
12.5%
18.8%
FLS 50
vPINN
2.1e-1
1.000
0.994
2.1e-3 0.069
0.069
1.1e-2
0.688
0.765
25.0%
18.8%
RoPINN
2.2e-5
0.022
0.039
1.5e-4
0.016
0.017
9.6e-4
0.173
0.197 74%
81.3% 74%
87.5 %
Promotion
99%
98%
96%
96%
84%
86%
99%
Vanilla
3.0e-6
0.015
0.030
1.4e-2
0.270
0,283
3.7e-5
0.023
0.027
PINNs-
gPINN
1.5c-6
0.009
0.018
OOM OOM 00M
3.7c-2
0.914
0.950
0.0%
0.0%
Former 58
vPINN
1.6c-4
0.065
0.124
4.5e-2
0.411 0.400
5.1c-5 0.016
0.022
0.0%
0.0%
RoPINN
1.0e-6
0.007
0.017
6.5e-3
0.165
0.172
1.2e-5
0.005
0.006 78%
100.0%
100%
Promotion
66%
53%
43%
54%
39%
39%
68%
78%
Vanilla
7.3e-5 0.031
0.061
9.2e-2
0.499
0.489
5.8e-2
0.922
0.954
gPINN
2.9e-4
0.030
0.061
2.6e-1
1.131
1.110
1.2c-1
1.006
1.041
31.3%
31.3%
KAN
VPINN
2.1e-1
0.998
0.996
9.0e-2
0.498
0.487
2.5c-2
0.853
0.853
43.8%
43.8%
RoPINN
4.9e-5
0.026
0.051
9.6e-3
0.177
0.191
2.2c-2
0.805
0.801
100%
93.8 %
Promotion
33% 16%
16%
89%
65%
61%
62%
13%
16%
As we stated in Table [ ID-Reaction and Convection are hard to optimize, so-called "PINNs failure modes" (24 33 In contrast, empowered by RoPINN, PINNs can mitigate this thorny challenge to some extent. Specifically, with RoPINN, canonical PINN (36), QRes [3] and FLS [50] achieve more than 90% improvements in ID-Reaction. Besides, RoPINN can further enhance the performance of PINNsFormer (58) and KAN (28), which have already performed well in DD-Recation or Convection, further verifying its effectiveness in helping PINN optimization.
Combining with other strategies Since RoPINN mainly focuses on the objective function design, it can be in- tegrated seamlessly and directly with other strategies. As shown in Table 3
Table 3: Adding RoPINN other strategies based PINN.
Time is for every 102 training iterations on ID-Reaction.
Method rMSE
ID-Reaction ID- DD-W Convection Time (s)
PINN (6
0.981
0.335
0.840
18.47
we experiment with the widely-used loss-reweighting method NTK [47] and data-sampling strategy RAR (51).
+gPINN 55
0.978
0.399
0.935
37.91
+yPINN 18
0.982
0.173
0.743
38.78
+RoPINN
0.095
0.064
0.720
20.04
Although NTK can consistently im-
+NTK 47
0.098
0.149
0.798
27.99
prove the performance, it will take ex- tra computation costs due to the calcu- lation of neural tangent kernels [[5]. Based on NTK, our RoPINN can ob- tain better results with slightly more time cost. As for RAR, performs unstable in different tasks, while RoPINN can also boost These results verify the orthogonal contribution and favorable efficiency of RoPINN w.r.t other methods.
+oPNN
0.052
0.023
0.693
29.96
+RAR 5
0.981
0.126
0.771
19.71
+RAR+ROPINN
0.080
0.030
0.695
20.89
8

---

4.2 Algorithm Analysis
Initial region size in Algorithm To provide an intuitive understanding of RoPINN, we plot the curves of training statistics in Figure 2 including temporally adjusted region size log( o ). train loss, and test performance. From Figure aa) we can find that even though we initialize the region size as distinct values, RoPINN will progressively adjust the trust region size similar values during training. This indicates that our algorithm can capture a potential "balance point" between training stability and generalization error, where the fluctuation of trust region size reveals the balancing process. Further, as shown in Figure Sauure2bbor 2b-c), if r is initialized as a value closer to the balance point (e.g. le-4 and le-5 in this case), then the training process will converge faster. And too large a region size (e.g. le-3) will decrease the convergence speed due to the optimization noise (Theorem
(8] Trust Region Size ?
(b) Training Loss Eq. (2]
(c) IMSE
1.-
0.-
inst moleeted
diby rited
miled
-10
Sattion Gol mi2e)
-15
-50
-45
-20
190 2 mo
1090
3.0
Training horations
Training Itoracions
Training Iterations
Figure 2: Optimization of canonical PINN 33 on the ID-Reaction under different region sizes. To highlight the region size change, we adopt the moving average over time and mark the temporal standard deviation with shadow. The steep training loss is caused by the learning difficulty of PDE.
Number of sampling points in Eq. 8 For efficiency, RoPINN only samples one point within the trust region to approximate the region gradient descent. However, it is worth noticing that sampling more points will make the approximation in Eq. ( more accurate, leading to lower gradient estimation error. Further, since RoPINN employs an adaptive strategy to adjust region Ω, a lower gradient estimation error will also make the optimization process adapt to a larger region size r. Therefore, we observe in Figure 3--bb that sampling more points will also increase the finally learned region size and speed up the convergence. In addition, Figure 3 shows that adding sampled points can also improve the final performance, which has also been theoretically justified in Theorem 3.12 that the upper bound of generalization error inversely proportion gradient estimation error.
[8] Region Size logt
(b) Training Loss Eq. (2)
(c) rMSE
1.0
-25
Single Ne Jungl 7g
Some Lane
02-
Semp 65 Parts
03-
406
1690
2203
Training horatiooss
Training Itaracions
Training iterations
Figure 3: Optimization of canonical PINN 336 the ID-Reaction under different sample points.
Efficiency analysis As discussed above, sampling more points can benefit the final performance, while we choose only sample one point as the default setting of RoPINN in the spirit of boosting PINNs without extra backpropagation promotion w.r.t. original PINNS (Table 2}. To provide a more comprehensive understanding of algorithm property, we plot the efficiency following observations. Firstly, computation costs will grow linearly when adding points. Secondly, more points will bring better performance but will saturate around 10 points, where the performance fluctuations of 9, 13, and 30 points are within three times the standard deviations (Appendix D.3)
gradient calculation, which has already achieved significant
efficiency-performmace performance curve in Figure 4 where
obtain the
Ablations To verify the effectiveness of our design RoPINN, we present ablations in Figure 5 is observed that although we only sample one point, even fixed-size region optimization can also boost

---

(a] PINN-R Reaction under various sample points
(b) PINN-ROPINN Wave under various sample points
Point setini
Parnay
0.07
moyy
0.08
0.06-
0.07
SOR
1944
05
200A
Poines
0.04
0.04
30 Pa
0.03
0.02
Pukio
125
100
150
200
Running Tirns per T00 Ianationa (4)
Running Time par 100 namaaiiona (4)
Figure 4: Efficiency and model performance w.r.t. number of samples, Note that the default setting of RoPINN is just sampling one point, which will not bring extra gradient calculation costs.
Vanila
trust Rog Calibration
Trust Rugion Calirocion
Region
92-
01
PMNN
Qifes
P15
PNNNsForrmrr KAN
PINN
Qlas
PLS
P. P.NWsFormar CAN
PMNN
Gites
FLS.
PINNSForrie KAN
(0) 1D-Reaction
(b) 1D-Wave
[o] Convection
Figure 5: Ablation study of RoPINN on different PDEs and diverse base models. rMSE recorded.
the performance of PINNs in most cases, demonstrating the effectiveness of introducing "region" to PINN optimization. However, as illustrated Theorem 3.9 the sampling process may also cause gradient estimation error, the relative promotion is inconsistent and unstable among different PDEs and base models. With our proposed trust region calibration, we can obtain a more significant and consistent improvement, indicating that achieving better balance between optimization and generalization (formally stated in Theorem 3.12) performs an essential role in training PINN models.
Loss landscape Previous research 24 has studied why PINN cannot solve the Convection equation and found that is not caused by the limited model capacity but by the tard-to-optimize loss landscape. Here we also provide loss landscape visualization in Figure 6 which is obtained by perturbing the trained model along the directions of the first two dominant Hessian eigenvectors [4 25
We can find that vanilla PINN optimized by PINN loss in Eq. @ empowered by RoPINN, the loss landscape significantly smoothed. This visualization intuitively interprets why RoPINN can mitigate "PINN failure modes". " See Appendix D for more results.
presents sharp cones. In contrast,
Convection
Error Map
Loss Landscape
Error Map
Loss Landscape
1.5
-23
0.0
-0.75
-0.7
21u
2u
(a) Vanilla PINN
(b) PONNN
Figure 6: Loss landscape of RoPINN and vanilla PINNs on the Convection equation. Error Map refers to the distance between model prediction and the accurate solution, i.c. (us u).
Conclusion
This paper presents and analyzes a new PINN optimization paradigm: region optimization. Going beyond previous scatter-point optimization, we extend the optimization from selected points to their neighborhood regions. Based on this idea, RoPINN is implemented a simple but effective training algorithm, where an efficient Monte Carlo approximation process is used along with trust region cal- ibration strategy to control the gradient estimation error caused by sampling, theoretically manifesting better balance of generalization and optimization. In addition to theoretical advantages, RoPINN can consistently boost the performance of various PINN models without extra backpropagation or gradient calculation, demonstrating favorable efficiency, training stability and general capability.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, and the National Engineering Research Center for Big Data Software.
References
Michael Francis Atiyah, Raoul Bott, and Lars Gårding. Lacunas for hyperbolic differential operators with constant coefficients i. 1970.
[2] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[3] Jic Bu and Anuj Karpatne. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving pdes. In SIAM, 2021.
[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2019.
[5] Gouri Dhatt, Emmanuel Lefrançois, and Gilbert Touzot. Finite element method. John Wiley & Sons, 2012.
[6] Lawrence C Evans. Partial differential equations. American Mathematical Soc., 2010.
[7] Bengt Fornberg. practical guide to pseudospectral methods. Cambridge university press, 1998.
[8] Olga Fuks and Hamdi A Tchelepi. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing. 2020.
[9] Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: Physics-informet geometry- convolutional neural networks for solving parameterized steady-stat pdes on irregular domain. Journal of Computational Physics, 2021.
[10] Christian Grossmann, Hans Hans-Görg Roos, and Martin Stynes. Numerical treatment of partial differential equations. Springer, 2007.
[11] Zhongkai Hao, Songming Liu, Yichi Zhang, Chengyang Ying, Yao Feng, Hang Su, and Jun Zhu. Physics-informet machine learning: A survey on problems, methods and applications. arXiv preprint arXiv:2211.0800 2022.
[12] Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi La, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, et al. Pinnacle: A comprehensive benchmark pdes. arXiv preprint arXiv:2306.08833 2023.
informed neural networks solving
[13] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In ICML, 2016.
[14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[15] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. NeurlPS, 2018
[16] Alan Jeffrey and Hui Hui Dai. Handbook of mathematical formulas and integrals. Elsevier, 2008.
[17] George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nat. Rev. Phys., 2021.
[18] Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. Variational physics-informed neural networks for solving partial differential equations. arXiv preprint arXiv: 1912.00777 2019.
[19] Ehsan Kharazmi. Zhonggiang Zhang, George Em Karniadakis. hp-vpinns: Variational physics- informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering, 2021.
[20] Reza Khodayi-Mer and Michael Zavlanos. Varnet: Variational neural networks for the solution of partial differential equations. In Learning for Dynamics and Control, 2020.
11

---

[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[22] David A Kopriva. Implementing spectral methods for partial differential equations: Algorithms for scientists and engineers. Springer Science & Business Media, 2009.
[23] Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett. Mitigating propagation failures in physics-informed neural networks using retain- retain-rssmmlee-reeeus (R3) sampling. In ICML, 2023.
[24] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physies-inforeed neural networks. NeurIPS, 2021.
[25] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural NeurIPS, 2018.
[26] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadencsheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR, 2021.
[27] Dong Liu and Jorge Nocedal. the limited memory bigs method for large scale optimization. Mathematical programming, 1989.
[28] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačić, Thomas Hou, and Max Tegmark. Kan: Kolmogorov-arnok networks. arXiv preprint arXiv:2404. 19756, 2024.
[29] Lobanova and Ataullakhanov. Running pulses of complex shape a Physical review letters, 2004.
model.
[30] Lu, Xuhui Meng. Zhiping Mao, and George Em Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 2021.
[31] Suryanarayana Maddu, Dominik Sturm, Christian Müller, and Ivo Sbalzarini. Inverse dirichlet weighting enables reliable training of physics informed neural networks. Machine Learning: Science and Technology, 2022.
[32] Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics-informed neural networks for approximating pdes. IMA Journal of Numerical Analysis, 2023.
[33] Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Lagrangian pinns: A causality- conforming solution to failure modes of physics-infommed neural networks. arXiv preprint arXiv:2205.02902, 2022.
[34] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison. Andreas Köpf, Edward Yang, Zach DeVito. Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[35] Maziar Raissi. Deep hidden physics models: Deep learning nonlinear partial differential equations. JMLR, 2018.
[36] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 2019.
[37] Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madelcine Udell. Challenges in training pinns: A loss landscape perspective. arXiv preprint urXiv:2402.011 018.0188,, 2024.
[38] Tomáš Roubsček. Nonlinear partial differential equations with applications. Springer Science & Business Media, 2013.
[39] Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 2018.
[40] Pavel Solín. Partial differential equations and the fintt element method. John Wiley & Sons, 2005.
[41] Thomas Stocker. Introduction climate modelling. Springer Science & Business Media, 2011.
[42] ENZO Tonti. Variational formulation for every nonlinear problem. International Journal of Engineering Science, 1984.
12

---

[43] Nobuyuki Umetani and Bernd Bickel. Learning three-dimensional flow for interactive aerodynamic design. ACM Transactions Graphics (TOG), 2018.
[44] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang. Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery in the age of artificial intelligence. Nature, 2023.
[45] Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality for training physics-informed neural networks. Computer Methods Applied Mechanics and Engineering. 2024.
[46] Sifan Wang. Shyam Sankaran, Hanwen Wang. and Paris Perdikaris. An expert's guide to training physics- informed neural networks. arXiv preprint (:Xiv:2308.0866. 2023.
[47] Sifan Wang. Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 2022.
[48] Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. JMLR, 2020.
[49] Ahdul Majid Wazwaz. Partial differential equations: methods and applications. 2002.
[50] Jian Cheng Wong, Chin Chun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.
[51] Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computer Methods in Applied Mechunicx and Engineering, 2023.
[52] Jiancong Xiao, Yanho Fan, Ruoyu Sun, Jue Wang, and Zhi-Quan Luo. Stability analysis and generalization bounds of adversarial training. NeurlPS, 2022.
[53] Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. Multiadam: Parameter scale-invariant optimizer for multiscale training of physics-informed neural networks. /CML, 2023.
[54] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. 2020 IEEE international conference on big data (Big data), 2020.
[55] Jeremy Yu, Lu Lu, Xuhui Meng, and George Karniadakis. Gradient-enhancee physics-informed neural networks forward and inverse pde problems. Computer Methods Applied Mechanics and Engineering, 2022.
informed
[56] Ya-xiang Yuan. A review of trust region algorithms for optimization. In Iciam, 2000.
[57] Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for nigh-dimensional partial differential equations. Journal of Computational Physics, 2020.
[58] Leo Zhiyuan Zhao, Xueying Ding, and B Aditya Prakash. Pinnsformer: A transformer- basdd framework for
neural networks. ICLR, 2024.
13

---

A Generalization Analysis in Section 3.1
This section will present the proofs for the theorems Section 3.1
A.1 Proof for Point Optimization Generalization Error (Theorem 3.3
The proof for the convex case is derived from previous papers 13 under the Assumption 3.2 We derive a more compact upper bound for generalization error in expectation for the non-convex setting.
Lemma A.1. Given two finite sets of selected points S = (a1,... ØN) and S' = -a',,... x'y) let S (x1,.... 2i Ey Bi41,'.. ,xx) be the set that is identical to except the i-th element, the generalization error in expectation is equal to the expectation of the error difference between these two sets, which can be formalized as follows:
N
Egen
Es,s'.A LA
E
C(A(S)),
1
E ((A(S),
(15)
N
Proof. Directly deriving from the in-domain loss, we have:
N
Es.A (S(())( = Es.A
C(A(S), xx))
N
Es .A
C(A(S)0), xt)
(Expectation on x²)
N
(16)
ES
Σ C(A(S), 21))
+8
Es.A (C(A(S)) [2) +
Then, according Definition 3.1
we have:
N Σ
N Σ C(A(S), 21)
Egen = 5 = ESS.'
C(A(S()),
(17)
N
N
1=1
Lemma A.2 (Convex ease). Given the stochastic gradient method with an update rule as Ga. (0)
0 a Vo.(0, æ) and L convex in 0, then for 3/ have (G,,,(00)) Ga (02)l |0 0g
Proof. For clarity, we denote g [VV x) TeL(02xx)]] Then we have:
a| (01) - Ga. (02) = |00 0₂ at (Vo((()) - Vo2,0,, x1)
|00 0g 2 2a (VoL(0), x) VoL(O,, x))² (0) (01-02) + a²g²
20 9 + a²9
< |10| 02
(Convexity and Assumption 3.2
3
2
||0 0g 2
(a S
(18)
Convex setting Next, we will give the proof for the convex case of Theorem 3.3 1).
Proof. According to Lemma A. we attempt to bound the generalization emor in expectation Egmn by analyzing the error difference between two selected sample sets. Denote and S' as two identical sample sets of size |S] except for one sample. Suppose that with the stochastic gradient method
14

---

on these two sets, we can obtain two optimization trajectories (0r) and (6 respectively.
According to Assumption 3.2 we have the following inequality:
E [1 æ) (u =]1] - LE [1|0, 8.1l]
(19)
We assume two optimization trajectories, both obtained under the same random update rule and
random permutation rule. Note that at the t-th step, with probability (1 - [S] ). the example selected
1
by the stochastic gradient method is the same in both S and S'. As for the other [ड] probability, we have to deal with different selected samples. Thus, according to Lemma A.2 we have:
1
E. [[|0+11 - 11111| = (1
JR [| (n)) - 2n
E [l (0 Ga (0))
[
[S]
$ (1
TB[10 - 0/1l0 +
E 110, - 0,11 + ||or VoL(up, Or VoVolug Il)ll] [
[
(20)
Due to the L-Lipschitz assumption of L, the gradient V VoL(ue, x) is uniformly smaller than L, then:
20,L
[[|0++1 - 1/+111 s E [[]0, - 0,1l +
(21)
In summary, since both optimization trajectories start from the same initialization, namely 0 8-- the following inequality holds:
2L S
[[ x) - [uugs m]]|
S
αt
(22)
[=1
From Lemma A.1 we have Egen
21.
11=1 Ot. [S]
Σ
Lemma A.3 (Non-convex case). Given the stochastic gradient method with an update rule as
Ga aVoL(0, x) then we have I[Ga, Ga, (02))l S (1 + a;3)l(0 02l1.
Proof. This inequality be easily obtained from the following:
(0:) - Ga. (02) = |0] - 02 a (V - VoL(O2,x))
|0] 021 + a: Vo
x))
S (1 + a;3) - 02/l.
(Assumption 3.
(23)
Non-convex setting Finally, we will give the proof for the non-convex case in Theorem 3.3 2).
Proof. We also consider the optimization trajectory (0,) and (0:)T_ from S and S', which
identical except for one element. We assume two optimization trajectories, both obtained under the same random update rule and random permutation rule. Let 8, = ||6, 0, and to {1..... ........S)) ,|8]} be a considered iteration. Here, < SS] because for L > (S), we must have di + 0. Then we have:
[[C(uar, x) L(ugr, m] = Po. = O)E || [[[uurr: x) L(ue,, x) 50 = 0]
- P(, + 0)E [| ([ x) [(u0,. ; x) )66,, + #0] P(6., # 00C
< LE [||@r 17
(Upper bound of L)
Ctu + LE [| - 0/ 6 = 0]
= [
(24)
15

---

Similar to the convex case, we analyze the expectation of parameter difference in the (t + 1) iteration as follows. Since a I 33 then we have:
2L
E [10++1 11 = 0] s (1
(1
E
111, - 8111
+
E 11
8,111 +
[
S
Bt|S
1
1
2L
51 +
t
t|S
Bt|S
2L
$ expl
7
+
(1+ exp(x))
1
3t/S
(25)
Accumulating the above in equations recursively, we have:
[||0T 04.1116rr =0 s
Σ
(
)
2L
BUS
t=t++
T
exp
(1
1
log
T
2L.
Σ k k=t+1
T
- log P
|
3t|S|
t
+1
T
2L
[8]
Σ
(1
1s) --1
S
t=ta+1
2L.
(1 13T
T (1- 15T) )
T
T3T
to
(Integral approximation)
B|S|
S
(26)
Organizing the above inequalities, we have:
2L
T
2L
E [|
07
0] -
3([S]
1)
1o
B(|S) - 1)
(27)
2L
T
2T.
B([
1)
B([S) -1)
According to Eq.
24 for arbitrary 2 1, we just choose lg = 1, then
C
+ (T -
[[[(ur x) Llug, m| S
(28)
S
3(S -1)
L
This boundary is tighter than 13 52 where the latter omits the -
21. term in Eq. 27
A.2
Proof for Properties of Region Optimization (Lemmma 3.4
Since gregion is defined as a region integral of L, Lemma 3.4 can be easily obtained by:
1
Bounded:
C Cregion (uo. x)) [2pl Jn. L(uo, E)dg max L(uo, x:)
8.00
(VoLregion (uepiionnu,, (z) Vel (1o,, T (0) - -0))
Convexity:
|22 Jo. (VoL(uo, E) VoL(uog x + E)) (0: 02)dE 2 0
Lipschitz:
|| (wa, x) - Liopion (usz: x])
(29)
[[[uu,, x + E) L(uu, E)] S 1| 021l
12 Ja.
Smoothness:
V Vo pregion 2:) - Vo (no.: x))l
< |22 Je.
[Vo[uu, x + +) x s 3 -21l.
16

---

A.3
Proof for Region Optimization Generalization Error (Theorem 3.5
Similar to the proof in Appendix A.1 we will discuss the generalization error on region optimization.
Convex setting Firstly. we would like to prove the convex case.
Proof. According to Lemma 3.4, the region loss also holds the convexity, Lipschitz and smoothness properties, which ensures that Lemma A.2 and A.3 still work for Leegion, We also focus the selected sample sets and S', which are identical except for one element. Thus, at t-step, the following equation satisfied:
[[|0+1 - 8/+1ll] =
E Grugion((e))
Grogion (6))1] + Gregion (0:) ai
Gregion "xe,
(8/)1)
|S]
01,2
[S] Greggion (0.)
$ (1
1
E [1|0 011l]
1
G "n (Di)l}
S
S
(30)
where Gregion (0) o Vo x). As for the second item the right part, also consider the upper bound V VgCCogion Cragion ug x). However, different from point-wise optimization, æ + Q2 could overlap with x' + S2,. For clarity, we define the overlapped area Sin. whose size is larger than zero when x' fall into the area centered at with size 2 [[2,). Actually, due to the boundary of Q we cannot always ensure (a' + 12,) C Thus, for simplification, we assume that the domain Ω can be projected to torus, where the out-of-domain samples will be re-included to Ω
Further, E9,, [am|=0 is simplified Eq,,=0 and Egg ef, s.t. [Pa|>> is shorted Ep,,.0 And the operator I is defined [[(::) max (0, min(1, (1) Thus, can obtain the estimation for the difference between the updated model parameters through the following derivations:
Gregion 2..
(0:)
Gregion "ay
(0))
2/d-11 [22]
En 0
[|Grogion (6,)
Gregion (#)]]
[22]
2(++1)
1pl
En
Crregion (0:)
Greglon ai
(0)1
[22]
[22]
2(d
Eng [110 -0'l + 20,
|22|
2((+1)
14rl
00. - ai [2, aa
1 / VoL(uo, x' + +E)dE
+1
En
Vol(ue, + E)dE ar "[2 a
[2
[
2(d-
Es [110-
8'11 + 20,
|S2|
2((+1))
Sprl
VoL(uo, E)dE at 12, / VoL(us, E)dE
+I
E
0z - 0g
Ok [22 Jm
[22]
2(d+1)
14r
|22
15
+I
Equ.o
201
[22
14
[92]
2(d
|
2(d+1) (
12p Smn
<
T
Rn,
[110e - 0/11 + 2. +I
Es 20 110 -0,1 +
20; To
[2
52
22
(2(0-1)(22,)
Din 12pl
E
[||0,
8,ll
2a,L
Ban >0
20rL
[22]
[2
E [1|0
0'1l 22,
|22]
(31)
In the above incqualities, the third inequality is based on the following derivations. Firstly, denote
=
[. Sa... VoL(ug, x + E)dg [o2.] San.. Vol(ugg, x + +)dE) |]. According to Assumption 3.2
17

---

we have
1m [2,1
3110; - 8:11 - 3|00 8:11- Thus, the following inequality holds:
1
VoL(us, x+ E)dg - at [tpl appl (
0r 8
at
Vol(un, x + E)dEE
1021 Am
110: 0111 20n9" (0: 01) + agg²²
(32)
< |0: 01112 2 ah g² + aggg²
2
(Convexity and g S B100 -6(1l
3
2 (az
< ||0 01112
3
Thus, recursively accumulating the residual at the 1-th step, we have:
[2r 2L
gen S (1
(33)
[
For the more general case, we no longer assume that the domain Ω can be projected to torus, resulting non-symmetric scenario when |m2ml O. This asymmetry arises due to the presence of boundaries, since points that could potentially intersect with the set æ + Sr may be truncated by the boundary. Specifically, we consider f = [0, (((+)) and Ω, [0, r] The concrete probability
P([Smnl > 0) and the expectation E Epn,./uu( ((2nn))))) can be calculated as follows:
r(1 - 3r) (d+1), E (( = 3(1 r)2
(31 4r) y(d+1)
P([2ml > 0)
(34)
(1 - r-
Thus, the general case of Eq. 31 can be reformulated using the identities above. We assume / < 0.5, as when { 2 0.5, it follows that P( > 0) 1. Specifically, Eq. (31) can be rewrite
Gregion (0x) 0
Grgginn ac
(0)]
P([2m
= 0)
||Grogle (f:)
Grteginn (63)1] + P([22n) > 0) "a,
[ | (0,) - Gregion "ac,
(6))
< P([2ml = D = [1|6 - -8'11 + L]
0g 0% - (ai a1 [22l nn.
/ Vel(ug, x + E)dE - at
1 Volluo xx + +E)dE) ||]
+ P([0ml > 0) En.,
[2p Srr
12, - |22iul
< P[[[mm] = 0) 18
1110
0'11 + 20, LJ + PP((2in)) > E)E
200 L
+ +([0mul > E >0
0, 0% - (au (a112.) Pim
VoL(ue, x + E)dE - -a113 |12,
VoL(uo, x + E)dE
Smn
14p11
102m 20rel
< 2((2ml O)LE =0 [110 - -0,1- + 20, L| + P([Sinl > 0)E 00 110
0'11
[2
naul
< E [110, 1II' L
p[[22 > 0)E >0
20x L
|,2,1
E (20(
E [10.
L. |1[1
P(
|52
(2)))-33)) 3rl 31 - 4r) ((84.))
E
[||6
2
1
3(1 - r)
(35)
Thus, recursively accumulating the residual at the t-th step, we have:
2 (21 - 331 - 4r) y(++1)
2
Egen <
Σ at
(36)
3(1 - r)a
S
Although the specific forms differ and the general case is much more complex, these two inequalities both share the same intuitive meaning: region optimization benefits from the overlap between æ + Ω and x' + Ω, Moreover, within certain range, the benefit increases as the value of becomes larger.
18

---

Thus, to keep the bound simple and easy to understand, the main text theorems are under the assump- tion that Q can be projected to a torus. Otherwise, 1a. should be replaced by (21-3r) (d+1).
[
3(1-))
Non-convex setting Next, will prove the non-convex setting. Similarly, assume Ω can be pro- jected to torus. Otherwise, the
[9.1| in the final bound should be replaced by (22 (3) ) yd=+1)
[[]
3(1-r)
Proof. For clarity, let M
12.11 [o]
Then, we can rewrite the Eq. 25 as follows.
If B(),) _ 24. we have:
1
E [100+1 - 1111 =
(1
I I110,
6ill] +
E Grregion (0.) Guestion (0))1]
[8]
[S]
(1
(1
E 1110:
8i1l + [S]
1
(1
M E 110. 6: + 2L
(1 - (1-M)
[8]
31.
1 -
2L (1 - M)
S + 1
]E[6d] +
1961
1 1-
2L
exp
Blde] + 3|| (1 M),
i 188
(37)
where the second inequality is based on the following derivations:
E
Gregioo (0t)
Gresgton (8))
[22]
2((--1)122
Eng
||Guorian (0.)
Gregioon a,.."'
(0)))
[S2]
2((+1)
12rl
+I
En,.0 Gregion (B:) - Gragion (8))
[22]
[22]
2(d
1) 1|22
Em [1|0 10 0'll + +2,
[22]
2(0+1)
12p1
11 -8,- -(a1 |12,1 ar
1 / Vollug, x + + ||]
+I
Ex
Vellus, x+ + - at |2 or
[22]
[
2(d-
(id-1)1121 1112
18 E 1110, 0'1l + +2,44]
[22]
2(4+1)
12l
/
1
/ Vol(ug, x + E)de)
|10 18-6pp 0'll + aa [22 Jan
Vol(us, x + + at 112,1 n
[S
2(d+1)
14d
|12p1 - 152gal 200
E
[221
[92]
2(d
1) |12
En, =0 [1 - 0% + L
[S2]
2((+1)
Min
a,,3100 - 0''l + [2 - 1Sin
+I
E ER
110 -0'1l +
20 T
[2
[2p
3 1
(2(++) [2,
1inll
E
0'll
JE [110, 0'1ll + 20,L
ER =0
201
[2
[
[2,1
1
|22l
<(1
+
[||0: EE
0'1l| + 2a, L(1
[2s
[22
M
+ 2L
(1
IE
[1I0,
0'1l]
3t (1 M).
(38)
19

---

Notably, E G (0t) Ge (0))l has an obvious upper bound, i.c. [ [E11 0
+ [t
21
And only when E(,,)= - 24 the bound derived by Eq. (38) is tighter. Furthermore, the condition that B(6,) v ب4 can be easily satisfied the beginning several iterations since (E()g) 0. Otherwise, we still take the following equation:
E |10 0/411166, 0] S exp (t
1 (181
2L
B[ +
(39)
BI|S
where we do not consider the benefits brought by the overlap area of + Ω2 and x' + S2
Suppose that at the first K steps E[őto+k] I 24, Accumulating the above in equations recursively. we have the generalization error bound accumulated to the first K steps as follows:
to K
K exp
exp
1- M- -
(1
(
(1-1s)
1
2L.
(1 M)
||S]
(
BUS
1=
ko+K
1
T
1 1- M
log 40 + K
2L
exp
(1
log
+ +(1
St|s] (1 M)
to + K
|S|
t
#=
to+K
Σ exp
(1
1
log
T
+
M
log
to + K
2L
(1 - M)
S
[
t
Bt|s
t=
(40)
ko+K
1
T
2L
10 K
exp
(1
log t
M)
S
St|S]
11
t K
Σ exp
(1
1
log
T
2L
Σ t=o++
to-K
exp
(1
log T
2L
(M²)
S
S1|S|
8
Bt|s
1=
to+K
T
2L
exp
(1
log
JM²,
t
St
+1
where J is finite value that depends on the training property of beginning iterations, namely K and
ty. The last inequality is from (fo* 18f s (1 + M), when |S] is sufficient enough.
Then, considering the all T steps, we have
E [||07 - 02 = 0] S
T Σ
mH
nK exp
(1
---
}
1
2L
/US
++++
<
exp
(1
1
log T
2L
JM
Σ k=t+1
1
S log 7
S
3t|s|
k
t=fo+1
T
2L
T
T3T
(1- ) 1-1 J
8|8
t=
2L
-11-ja) (d)) - _T--T111
(1- -T- 1ht))
40
JM
(Integral approximation)
A|S
[
(41)
Thus, following similar proof process as Theorem
3.3 2), we can obtain:
2L
2L
E [|0T 11 = 0] <
to 3(S) 1) - 2
(42)
B(S
With 4 = 1, we have the generalization error under the non-convex case satisfies:
C
2L - 1) JLM2
E. [[C(uor ax) - L(wos m1 S
(43)
S
3(S - 1)
20

---

A.4 Proof for High-order Constraint Optimization (Corollary 3.6
First, we would like to prove the following Lemma.
Lemma A.A. Suppose that & is bounded by C for all 0, a and is L-Liipcciitz and 3-smooth for 0, then the first-order j-th dimension loss function 0 Cregion is also bounded by for all 6, and is Lipscci and 23-smooth for 0.
dzz
Proof. For the bounded property, with the non-negative property of loss function, we have:
a
/ C(uo, x E)dE JAL/aa
L, Eregion (uo: x)) dz,
[(ug, x + E.) [(uo, x + Eo)dg S C,
dz,
(44)
where &, = (...,,,.. E 24 (2y and Eo = ( ,,, ....).)................. E 22 (z-- As for the Lipschitz and smoothness, we can obtain the following inequalities:
a
Lipschitz:
||
Cr Or,
Cregion
(uBl x)
C,
Cregion (ugz x)ll
/ Jnilas
|1
[(us, + Er)
L(ug, Eu) [(uez x + E,) [(vez, x Eo)dEll
$
[[[(u], x + Er)
[(us2 x + E,)l + [[[s, x + Eu) L(ug, x + Eo)lldE
50, \az
S 2T 02ll
θ Cregion (10,; x)) Vo Oz,
Smoothness:
Vo 0zz
pregion (102: x)]
/
= =|Vo
C(uo, x + +E,) L(uo, x Eo) L(ues: x E,) [(ue, x + En)
$ Ja
[Vo[(uo, x E.) VoL(uog x E,)
+ RAA=
I[VoL(us, x x+ En) VoL(uo, + Eo)lldE
< 2, 0211-
(45)
Thus, 0z,
0 gregion also bounded by for all 0, x and is 2i-Lipschitz and 2,3-mmooth for 0.
Next, we will give the proof for Corollary 3.6
Proof. According to Lemma A.3 we have the gradient update operator G
rrgionna
for Cregion Ozg
4,
satisfies the following inequality:
Gayz
Gaegion,
(01)
Go
region..x
(02) < (1 2a,,3) 10: 22ll
(46)
Let M
10_l [2 since 0g S
1.
we can rewrite the Eq. 37p as follows:
23r
[10 01+++1 8% 16, = 0]
2L E 110, 0,11
2/ (1 - Mn)
(47)
< (1 S (1 + E 1110, 6,11| +
(1
S
where the second term derived from Eq. 38
by substituting to 2T. and B to 2,9, which
E
Gar.
region.23
(0.)
G region.2) crc.a""
(0t)
< (1
2
E [110:
0, + 40,L(1
12
S2
(48)
M
2L
= (1 +
E [II,, o'll +
31 (1 M).
Thus, following the same derivation Theorem 3.5
we have Corollary 3.6 holds.
21

---

B Algorithm Analysis in Section 3.2
This section contains the proof for the theoretical analysis of our proposed algorithm in Section 3.2
B.1 Proof for Convergence Rate of RoPINN (Theorem 3.8)
The crux of proof is to take expectation for Monte Carlo sampling.
Proof. From Taylor expansion, there exist x' such that:
C Ciotion (18 2t
L, L,egion (uo, az VoL us. + x))
- (ue, x) at V C ug x + E) T V V Cregion (uo,, x)
2 (as VoL(uu, 2)) V rregion (10 x (ar ;
1
(49)
at V VoL(uo, uq, 5) V V Crngion (uo v) oiL2H
C. Cregion (uo:: (uo.; Φ)
2
Taking expectations to & on both sides, since E V Luu x + E)) = V e
Cragion [ug, x + E), we have:
(un, x)) 04 VoL(us, x + E) T V V Cregion (uo,: x) + a LLLL H
E [[_
Cregion (10
x.)
]
S E
c
cregion
2
of 1.2
E
Creiion (ug, x;)
a,E VoL Cregion (uo, x)) |)
2
(50)
Rearranging the terms and accumulating over T iterations, have the following sum:
Σ O₂ V V
Cregion (ug,
)1
S
E [
Cregion (ug, x]] E [ [rogion (1g (1
T-1
a.
L2
(=0
(uoT: x)) + L T-1
S c Cregion
(uoo x))
L.
Eregion
T L
< Cregion (uso: x))
Pregion (u. x) + 2
(51)
where u, represents the global optimum. Here we run the gradient descent for random number of iterations 7. For r = iterations with probability:
P(r=
(52)
Thus, with or V2+I* we have the gradient norm is bounded by:
E
Vo (wo, m11l
Rr
V 0
Gegion (wg, 1² [|]]
region
L2H T-1
Cregion (en x)) Cregion (Ue, xt) +
ai
2
1=
L2H
(2VT)-
Crogion (uer x)) Crogion (ur, x))
log(T + +1)
2
0
(53)
22

---

B.2 Proof for Estimation of RoPINN (Theorem 3.9
As presented in Eq. 88) and (50). we approximate the region optimization with the Monte Carlo sampling method. For better efficiency, we propose only to sample one point at each iteration. However, this will cause an estimation error formalized in Theorem which can be directly derived by the definition of standard deviation follows:
Proof. According the definition of 2. Cregion in Eq. we get
(g,)( [IVoL x + + V VoLr Cr ug x)) 2
27
+ Vo |2,1 / C(uo, E)dé
1
(54)
=Ee(((2.))
=1loe-e((.)) (VoL(us, E))Il
B.3
Proof for Estimation of Trust Region (Lemma 3.10 and Theorem 3.11
First, give the proof for Lemma 3.10
Proof. According to Assumption 3.2 there exist x', such that the following equation holds:
VaL(ua, 21) - V Lus, 22)
VoL(uo, 21)
(55)
x VoL(us, z2) + a-- aa- VoLuug x2) V2 (1
Thus, the following inequality holds:
VoL(uo, 21) - V VoL(ug..1 22) - 21) - Vol(ug, 22)1l
(VoL(u) - VoL(ue, 1: 22) (VoL(uo, 21) VoL(uo,
(56)
| VoL(un. L 22) V3 VuL(um 11 2')l
BLa-11
Next, we will prove Theorem 3.11
Proof. This theorem can be proved by demonstrating that: for all i, E (1, ;
lim VoL(ue 24) = VeL(up, 24)
1o
(57)
lim (Vo 2,) Vol(ue 24)) VoL(ue, 24) VoL(ue, 2z).
For the first equation, since + O, given Vc, there exists a constant M such that any t > M, ai I LoLa- Thus, for any > M and any i,j (1,.... ,T),, the following equation satisfied:
VoL(us, m.) voL(uo, 24)1
VoLLuu VoLluo,
(58)
Ot o S
Thus, lim Vol(ue 2,) VeL(up, ug, 2). Therefore, given Ve', there exist a constant M'' Vt > M', V 24) - VeL(us, 2:) 5.
As for the second equation, for any 6 M', the following equation is satisfied: Vo xi) - V VoL(uo, 23) - VoL(uo, - voL(uo,, 2;)l|
(59)
< 7o VoL(us, : +
25) VoL(up, 2;)1 S <
Thus, Theorem 3.11 can be proved by replacing the gradient of past iterations with their limitations.
D
23

---

B.4 Proof for Region Optimization with Gradient Estimation Error (Theorem 3.12
Convex setting Firstly, we would like to prove the convex case as follows.
Proof.
Similar to the proof of region optimization in Appendix A.3 at the t-th step, we can obtain the following equation:
E [10 - [[+2l]] = (1
E
[I|
G
(appro((0))
(6.)1] +
E Garpprx (0.)
(0))l
[
[8]
$ (1
1
E ||0; B'll]
1
Gapprox 01
(0.)
Gupprox as,
(0))
S
[
(60)
where Gapprox (0) V o Capprox (0, x))
VoL((0 + E) U(2,). Suppose that
have sampled E Q,, the second term on the right part can be bounded as follows:
E
|| Gapprox (0.) - Gapproo (8)) ae
ai x'
$ E [[]0: 04 Vo Vo x + E) 8¹ - at VoL(o, x' + E']]]
S E [110, ar V Cregion (0,, x) 8 V L Lregion (6', x I]]
E [||aaA VoL(Or, E) 04 V of (0:: 2
(61)
E [||a, VoL (0000,, 0,, + +²) at V Vo Cregion (0); x
12r
S E [100, - || + 20, [1
+ EokEr.gra
(Based on Eq. (31
|22
[22
E [1 8|ll + 20 [((
+ Erggrad
||2]
Thus, recursively accumulating the residual the (-th step, have:
[0p 1
2L
T
Eyon S [(11
Eygraad
ar-
(62)
[
Non-convex setting Similarly, we can prove the non-convex setting as follows.
Proof. It is easy prove that Lapprox (0, x) is still T-LLpphitz---smmottness for 0. For clarity, define that M
[h.1 Thus, based on Eq. (38) we have the following derivations.
[52]
If ((6))) <
2L 2 E grgaa, we have:
8 BM
E
|110+11
|1000 0442
0]
1 / EE [11e: 0,11]
- (1
(1 +
+
[S] E
G[pppprox (8.)
Coppprox an
(8))
|S|
1
<
(1 } L 1110 0,11]
[8]
+
[E
Gregiion (0.) Gregion (0)1]
[
1
(63)
+
EE
[la + E) as VoL Cregion (,,, x
[S]
+
EE
[Illav VoL(O.. x' + - V rregion (0, x [I]]]
[
1 M
20
<
(1
+
[[(b]] +
(L(1 M) + Er.grad)
t
S
1 1-M M
< exp
[| S
(L(1 - M) + Ey Er.grad)
7
24

---

Otherwise, we still consider the following inequality:
E [10 111111 = =0] 1 exp (: t|s
1
27.
El6g]
+
(64)
St|S]
Suppose that at the first K' steps E[ +x 1 S
2L 2
BM gradd
Accumulating the above in equations recursively, we have the generalization error bound accumulated to the first K' steps as follows:
fo
Io exp
(1-11 1- M
1-M
II 1 exp
(1-15)
2
(L -M) + Er.grad)
||51
1151
BI|S|
1=
to
T
1 M
to + K'
2
exp (1
log
+1
log
(L M) + Er,graa)
[
to + K
[
1
to
T
M
to + K'
2
exp
log
log
(L(1 M) graad
S
|
St|S
K
T
2
to + K'
exp
(1
log
(L(1 - M) Engrad)
8
St|S|
t
1=
K"
exp
(1
log
T
2L
Ka+K"
Σ
exp (1-
1
log 1
T
2 (LM² + Engrad + M))
8
S1|S|
S
Bt|s]
1=
t=o++
T
2L
exp
(1
log
J'LM² + J' J'Engrad (1 + M),
S
St|S
1=1++1
(65)
where J' is finite value that depends the training property of beginning iterations, namely K' and to. The last inequality is from (fo
I (1 + M). when (S] is sufficient enough.
Then, considering the all T steps, we have
[||0r - 01.11/rr =0) 0]
A +
Σ
T
rT.
Txxp exp
1
)
2L
t|S
Bt|si
k+o+
Σ
T
cxp
(1
log
T
2L
J'M J' grar (1 + M)
T Σ k=t+1
s log 7)
S
t
Bt|S
t=
2L
T
137
Σ
(1
) - J'M + J E'Erggra M)
3|S|
(=t++1
$
2T.
T--
[8]
1
to
) T -1 (b)))
J'M J'E Er.gra M). (Integral approximation)
||S|
S
(66)
Next, following the proof in Appendix A.3 A.3 we can obtain the generalization bound as follows:
2T - 1) J'LM22 J' Erag (1 + M). B(S) 1)
[[C x) L(ug, || S
(67)
S
Note that K" does not exist when 2L < that the region size too large and brings serious gradient estimation error. Introducing "region" cannot bring better generalization bound in this case.
3 GM E Engrad, then J' 0, which corresponds the situation
25

---

C
Implementation Details
This section provides experiment details, including benchmarks, metries and implementations.
C.1 Benchmarks
To comprehensively test our algorithm, we include the following four benchmarks. The first three benchmarks cover three typical PDEs (plotted in Figure which widely used in exploring the PINN optimization [24] 37). The last one an advanced comprehensive benchmark with 20 different PDEs. Here the details.
2.55
03
2.50
7:25
-0.0
-10
(a) 1D-Reaction
(b) 1D-Wave
(c) Convection
Figure 7: Visualization of the solution u for the first three benchmarks.
ID-Reaction This problem one-dimensional non-linear ODE, which describes the chemical reactions. The concrete equation that we studied here can be formalized as follows:
du - pu(1 - u) 0, x (0,2x), t (0,
Ōt
(x T²
(68)
u( 0) exp
2( /4 x [0,2m],
u(0, u (2T,, (), E [0, 1].
The analytical solution to this problem u(x t) = h( and h(x) exp
h(x))
fx--a)
In
2/r
experiments, set p This problem is previously studied as "PINN failure mode"
[24]
which is because of the non-linear term of the equation (29). Besides, as shown in Figure a),
contains sharp boundaries for the center bigh-value area, which is also hard learn for deep models.
Following experiments in PINNsFormer (58), we uniformly sampled 101 points for initial state Oo and boundary 0R and uniform grid of 101 x 101 mesh points for the residual domain f. For evaluation, we employed a 101 101 mesh within the residual domain Ω. This strategy is also adopted for Wave and Convection experiments.
This problem presents a hyperbolic PDE that widely studied in acoustics, electromag- netism, and fluid dynamics [T. Concretely, the PDE can be formalized as follows:
IDve
pu
8u
0, x (0, 1) t (0, 1),
dr
u(x, 0) (in() + sin
[0, 1].
2
(69)
Ou(u, 0)
0, [0,1]. 1].
0t
w(0.t) u(1,t) 0, [0.
The analytic solution for this PDE is u(æ t) = sin Tσ) cos(2 + _/2 sin(3rr) cos(2,7t), We set 3= for our experiments. As presented in Figure (b). the solution is smoother than the other two datasets, thereby easier for deep models to solve in some aspects. However, the equation contains second-order derivative terms, which also brings challenges in automatic differentiation. That is why gPINN [55) fails in this task (Table 2).
26

---

Table 4: Details of datasets in PINNacle [2] (16 different PDEs included in our experiments), including the dimension of inputs, highest order of PDEs, number of train/test points and concrete equations. Here we only present the simplified PDE formalizations for intuitive understanding. More detailed descriptions of PDE type and coefficient meanings can be found in their paper [2].
PDE
Dimension
Order
Nurain
Nies
Key Equations
1d-C
ID+Time 2D+Time
16384
12288
Burges
8y. v Vu vAu=
2d-C
98308
82690
24-C
2D
12288
10240
Au 0
2d-CG
2D
12288
10240
Au + k² = f(x, 4)
Poisson
3d-CG
3D
2
49152
40960
n kfu f(z, 2), 1,2
2d-MS
2D
12288
10329
(a(x)) Vu) = far, y)
2d-VC
2D+Time
65536
49189
V(((>)) Vu) = f(z,
Heat
2d-MS
2D+Time
65536
49189
du 37
+/2 uyy =
[500m]2
2d-CG
2D+Time
65536
49152
One
Au = 0
NS
24-C
2D
2
14337
12378
u Vu + Vp
1 Re
Au
2d-CG
2D
14055
12007
1d-C
ID+Time
2
12288
10329
Us 4u g 0
Wave
24-CG
2D+Time
49170
42194
22
u(x, 1)
[(5) 57
Chaotic
GS
2D+Time
2
65536
61780
u:
E1 Au bb1 u) uv
V: = E de + u
High- dim
5D
49152
67241
Au x²
0u 4 Σ. sin xx)
HNd
5D+Time
65537
49152
kAu + f(x
This problem is also hyperbolic PDE that can be used to model fluid, atmosphere, heat transfer and biological processes 41.. The concrete PDE that we studied in this paper is:
Convection
du
+ 3 du
= 0, x E (m,, (0,
0t. 01
(70)
u(x, 0) = sin(x), x E [0,2 2n],
m(0, () u22 1), 1 [0, 1].
The analytic solution for this PDE is u(x, t) = sin(x 3t), where 3 set as 50 our experiments. Note that although the final solution seems to be quite simple, it is difficult for PINNs in practice due to the highly complex and high-free frequrryy patterns. And the previous research [4] has shown that the loss landscape of the Convection equation contains many hard-to-optimize sharp cones.
This benchmark 12 is built upon the DeepXDE (30, consisting of wide range of PDEs and baselines. In their paper, the authors included 20 different PDE-solving tasks, covering diverse phenomena in fluid dynamics, heat conduction, etc and including PDEs with high dimensions, complex geometrics, nonlinearity and multiscale interactions. To ensure comprehensive evaluation, we also benchmark RoPINN with PINNacle.
PINNacle
During our experiments, we found that there are several subtasks that none of the previous methods can solve, such the 2D Heat equation with long time (Heat 2d-LT), 2D Navier-Stokes equation with long time (NS 2d-LT), 2D Wave equation with long time (Wave 2d-MS) and Kuramoto-Sivann Sivashinskky equation (KS). In addition the challenges of high dimensionality and complex geometry mentioned by PINNacle, we discover unique challenges in these tasks caused by long periods and high-orde derivatives of governed PDEs, making them extremely challenging for current PINNs. To solve these problems, we might need more powerful PINN backbones. Since we mainly focus on the PINN training paradigm, we omit the abovementioned 4 tasks avoid the meaningless comparison and experiment with the left 16 tasks. Our datasets are summarized in Table
27

---

C.2 Metrics
In our experiments, we adopt the following three metries. Training loss, rMAE and rMSE. And the training loss has been defined in Eq. (2). Here are the calculations for rMSE and rMAE:
rMAE: V Eres (u)(z) (x))
EmEs (uo(z) (x))
rMSE:
(71)
EmeS |u, (x))
Exes (u))2))²
where u. denotes the ground truth solution. Note that the model output and ground truth can be negative and positive, respectively. Thus, these two metries could be larger than 1.
C.3
Implementations
For classical base models PINN [36]. QRes 3 and FLS [50], we adopt the conventional configuration from previous papers (58). As for the latest model PINNsFormer (58) and KAN (28). use their
official code. Next, we will detail the implementations of optimization algorithms.
As we described in the main text, we the initial region size r= 10-4, past iteration number To E (5, 10} and only sample point for each region at each iteration for all datasets. The corresponding analyses have been included in Figure for r, Figure 3 for sampling points and Appendix D.I for To to demonstrate the algorithm property under different hyperparameter settings.
RoPINN
In addition, formalization for region optimization in Eq. 8 only involves the equation, initial and boundary conditions, where we can still calculate their loss values after random sampling in the extended region. This definition perfectly matches the setting of ID-Reaction, DD-Wave and Convection. However, PINNacle [12]. some tasks also involve the data loss term, such as the inverse problem (Appendix D.2), which means we can only obtain the correct values for several observed or pre-calculated points. Since these points are pre-selected, we cannot obtain their new values after sampling. Thus, we do not apply region sampling these points in our experiments. Actually, the data loss term only involves the forward process of deep models, which is pure data-driven paradigm and is distinct from the other PDE-derived terms in PINNs. Therefore, the previous methods, gPINN and vPINN, also do not consider the data loss term in their algorithms.
gPINN For the first three benchmarks, we add the first-order derivatives for spatial and tempo- ral dimensions as the regularization term. We also search the weights of regularization terms in {1, 0.0,0.011 } and report the best results. As for the PINNacle, we report the results canonical PINN following their paper [2] and experiment with other base models by only replacing the model.
vPINN We follow the code base in PINNacle, and implement to the first three benchmarks. The test functions set as Legendre polynomials and the test function number is set as 5. The number of points used to compute the integral within domain Ω is set as 10, and the number of grids is set differently for each subtask, with values of (4, 8, 16, 32) for PINNacle and the same other baselines for the first three benchmarks.
Other baselines In Table 3 of the main text, we also experiment with the loss-reweighting method NTK [4 and data-resampling method RAR [51]. For NTK, we follow their official code and recalculate the neural tangent kernel to update loss weights every 10 iterations. And the kernel size is set as 300. As for RAR, we use the residual-bate based adaptive refinement with distribution algorithm.
D Additional Results
In this section, we provide more results as a supplement to the main text, including additional hyperparameter analysis, new experiments and more showcases.
D.1 Hyperparameter Sensitivity on To
As we stated in Algorithm| [ we adopt the gradient variance of past To iterations to approximate the sampling error defined in Theorem 3.9 In our experiments, choose To from (5, 10). which can
28

---

624-
031-
0.4
MB
032
546
0.4
548-
100
080
(a) 1---Recction PINN
[b) 1D- PINN
(c) Convection PINN
(d) Convaction PINNsFormer
Figure 8: Hyperparameter analyses for To in RoPINN based on PINN 366 and PINNsFormer 158]
on different benchmarks. We change To in {1,5, 10, 15, 20, 25, 30 and record the rMSE.
achieve consistently good and stable performance among different benchmarks and PDEs. To analyze the effect of this hyperparameter, we further add experiments with different choices in Figure 8
As shown in Figure 8 we can find that under all the choices in {1, 5, 10, 15, 20, 25, 30), RoPINN performs better than the vanilla PINN. Specifically, in both ID-Reaction and ID-Wave (Figure (a-b)),
the model performs quite stable under different choices of To- As for Convection in Figure 8 c-d). the influence of To relatively significant in PINN. This may caused by the deficiency of PINN in solving Convection, where all the PINN-based experiments fail to generate an accurate solution for Convection (rMSE>0.5, Ta If we adopt more powerful base model, such as PINNsFormer [58] this sensitivity will be alleviated. Also, it is worth noticing that, even though in Convection, RoPINN surpasses the vanilla PINN under all hyperparameter settings of To-
Besides, we can observe that the model performance slightly decreases when we set To with a relatively large value. This may come from the difference between parameters 0, and 8,429, which will make the gradient variance approximation less reliable (Eq. (58) in the Theorem proof).
D.2 Experiments with Data Loss (Inverse Problem)
As stated in the implementations (Appendix RoPINN can also be applied to tasks with data loss. Here we also include inverse problem in PINNacle to testify to the performance of RoPINN in this case, which requires the model reconstruct the diffusion coefficients of the Poisson equation from observations on 2500 uniform grids with additional Gaussian noise.
Table 5: Experiments on the Possion in- verse problem (PInv) of PINNacle.
Method
rMAE
rMSE
PINN 33
7.3e-2 7.30-2(002%)
8.2e-2
+gPINN 555
8.0e-2((18)
*vPINN
18
1.3e+0
1.8c+0
RRoPNNN
6.7e-2(8.8%)
7.3e-2(11 4%)
As presented in Table 5 in this task, RoPINN can also boost the performance of PINN with over 10% in the rMSE metric and outperform the other baselines (gPINN and vPINN) that cannot bring improvements. Note that in this experiment, we failed to reproduce the performance of vPINN reported by PINNacle. Thus, we report the results of vPINN by directly running the official code in PINNacle.
D.3 Standard Deviations
Considering the limited resources, repeat all the experiments on the first three typical benchmarks and our method on the PINNacle three times and other experiments one time. The official paper of PINNacle has provided the standard deviations for PINN, gPINN and vPINN on all benchmarks.
We summarize the standard deviations of PINN in Table 6 As for other base models, the standard deviations of FLS, QRes and KAN are within 0.005 on ID-Wave and Convection, and within 0.001 for I DD-Reaction. PorNersrmee's standard deviations smaller than 0.001 for all three benchmarks.
Table 6: Standard deviations for canonical PINN three typical benchmarks. The confidence for RoPINN achieving the best performance is over 99% in all three benchmarks.
Stadaard Deviations
ID-Reaction
ID-Wave
Convection
PINN
36
0.981+55e-4
1.335+1e-3 0.399-33e-3
0.840++55-4
+gPINN 55
0.978+33e-4
0.935++3e-3
+vPINN 18
0.982+33c-3
0.173++1e-3
0.743+220-3
+RoPINN (Ours)
0.095118e-4
0.064:11e33
0.720:12e-3
29

---

D.4 More Showcases
As a supplement to the main text, we provide the showcases of RoPINN in Figure 9 showcases, observe that RoPINN consistently boost the model performance and benefit the solving process of boundaries, discontinuous phases and periodic patterns.
From these
Ground Truth
PINN Error
PINN-RoPINN Error
PINNsFormer Error
PINNsFormer- Error
1D 1D-Reaction
.111
1D-Wive
Convecttio
632
633
825
625
665
4.0
08
.220
Figure 9: Showcases of RoPINN on the first three datasets based on PINN and PINNsFormer.
D,5 Experiments with Advanced Quadrature Methods
In our implementation, RoPINN employs a simple Monte Carlo sampling to approximate integral. Obviously, we can adopt more advanced quadrature methods, such as Gaussian quadrature [16). Thus, we also experiment with 2D space Gaussian quadrature, which requires square num- ber points and the one-point-sampling situation will degen- erate to center value. As shown Table we can find that under our official setting (only sampling one point), Monte Carlo better, while Gaussian quadrature is better in more points. Note that although Gaussian quadrature has the potential to achieve better performance, sampling more points may contradict our motivation of boosting PINNs without extra backpropagation gradient calculation. Thus, choose the Monte Carlo method, which works better under high-efficiency settings.
Table 7: Comparison between Monte Carlo approximation (Monte Carlo) and Gaussian quadrature (Gaussian) on ID Reaction. rMSE is recorded.
PINN+RoPINN
Monte Carlo
Gaussian
Sample Point 0.095
0.109
Sample Points
0.066
0.059
Sample 9 Points
0.033
0.030
E Full Results on PINNacle
In of the main text, due to the context limitation, we only present the proportion of improved tasks the total tasks. Here we provide the complete results for 5 based models for PINNacle (16 different tasks) in Table and Table 9 where we can have the following observations:
RoPINN presents favorable generality in varied PDEs and base models. As we described in Table 4 this benchmark contains extensive physics phenomena. It impressive that our proposed RoPINN can boost the performance of such extensive base models on wide scope of PDEs. highlighting the generalizability of our algorithm.
RoPINN is numerically stable and efficient for computation. As training paradigm, RoPINN does not require extra gradient calculation and also does not add sampled points, which makes the algorithm computation efficient. In contrast, other baselines may generate poor results or encounter NaN or OOM problems some PDEs.
30

---

Table 8: Full results of gPINN (55.
vPINN 18 and RoPINN under different base models on
PINNacle [12] (16 different PDEs). A lower rMAE rMSE with higher relative promotion indicates better performance. The promotion over vanilla is recorded in parentheses. For clarity, we highlight the value with blue if it surpasses the vanilla PINN, gray if it fails (over 10 times worse than the vanilla PINN), or is numerically unstable (NaN) or out-of-memory (OOM).
Vanilla
gPINN
55
vPINN 18
RoPINN (Ours)
(Part
PDE
iMAE rMSE
IMAE
iMSE
iMAE
iMSE
iMAE
rMSE
Id-C
1e. 3.3e-2
3.7e-1
5.1e-1
4.02-2-372.29)
3.5e-1 (-952.3%)
9.15-3(15.8))
1.46-2(5668)
Burges
2d-C
4.5c-1 5.2c-1
(.97-1(8.79)
5.4--1(3.8%)
6.6e-11-46.45
(4.09)
(1.3-1().3%)
2d-C 77.5e-1 6.8c-1 |
7.70-1(33096)
7.0c-1(4400))
4.6c-
4.9e-1(27.4%)
(44.99))))))
(225-1(2.5%))
2d-CG
5,4e- 6.6ie-
1.37-1()7.11))
7.9e-1(22100 (9)
(,47-1
(.96-1(56.4%)
(24.11)))15)
(..1-1
Poisson
4,2e-1 5.0e-1
5.2e-1(33.5) 4.36-1(4.2%)
3.00-11 (c-1(99.4%)
7.4c- ((-677(-)
4.7e-1c-11998
(.67-1(.7%)
2d-MS 6.4c-1
6.7e-1(13.28)
6.2e-1(2.9%)
9.66-16/3.698)
9.70-1652.115)
7.74-1(0.46)
(.43-110.3%)
PINN 36
2d-VC]1.2c+09.55
1.9+11
1,6++1
8.8c-1
9.4c- (4..5)))
(27.69))
(19.91(1977%)
Heat
4.7e2 6.9c-2
1.0c+0
8,5c-1
9.3e-1
9.3e-1
4.44-2(6.49)
3.41-2/51.19)
2d-CG/ 2.3c-2
(.9--1 2.10)
3.1e+0
9.3c-1
1.50-2(43.5%)
2.0c-2 (12.66)
2d-C le 5.1e-2
46- 4el-11-998.685
4,
4.9e-1
2.0e- 22522229)
2.9e-1c-478.
4.1.-202.2%)
(12.-2(16.15)
NS
2d-CG 2d-CG 1.8e11
1.1c-1
4.20-
2.9--1 (-167.0%)
9.9e- (544446))
9.9e- (-812.29)
1.5e- (19.0%)
9.80-2(10.29)
1d-C 5,55c1 5.5c-1
7.0c-1(2700
7.2c- (x-11999))
1. 1.4c+0(-11..9) 4c-11333.59
3.8e- (21119)
(22.-1()800))
Wave
20-CGp2.3++01.6em 000ee+001.ee1 9.9ee000099111111.. (56.6))
1.0e O0337.99))
(.0.-1))
(.9.81(6888%)
(51.3%))))%
Chaotic
GS 1e-2 9.4e-
9.5e-2(-109)
8.9e-1
1.2e+0
2.1e-2(218)
9.3e-2.0.45)
High-
PNd 3.2c-3 1-
(6-3--11.55)) 2.7e-36-13.78)
NaN
NaN
6.7e-4(2.95)
6.4c-4:13685)
dim
HNd
1.2e-2 5.3e-3
3.60-3(710%)
4.6e-303.6%)
NaN
NaN
5.6e-4195.59)
7.3c-4.86.%))
Proportion of improved tasks
18.8%
18.8%
25.0%
25.0%
93.8%
100.0%
Id-C (5.8e-3 2.0c-2
3.6e-1
5.1e-1
2.0e-2t.24.... 8. Je-2:309..30 396)
5.74-3(1.49)
1.8e-2(6.89)
Burges
2d-C [3.2e-1 4.8e-1
(.98-1(35319)
5.4--1611199)
1. 1ele002266.98)
1.3e+00-161.22 16).2))
3.20-1
4.78-1(1.85)
2d-CC 1,9e-1 6.8e-1
1
66- (6-5944)))
(.2.-1()599))
3.0e-1(55.6%) 3.77-167
7.0e-1e3.770 2.96-1(066))
Ed-CGG..1e1 7.4c-1
(.36-1(135699))
7.80-1(6..0%)
6.4c-1c:188.95
7.3e- ((0.4)))
2.9c-1(4.0%)
7.25-1(2.5%)
Poisson 3d-CG
1.96-113558.89)
6.4.-11-10.7%)
8.0c-16290.45)
7.4c-1((28.4%)
8.9e-21
5.6e-113.5%6
20-MM|]..e+077.e-
(.5
5.0e-1(36.59)
9.7e-1611888
9.8e-1(-24.45)
1.9400-13.19))
QRes
2d-VC[22.0e-1 1.3e+0
7.2c+0
5.9e+00636..0%s)
3.3e-1
(.66-1(99695) (.29-1(74995)
1e1++0(1.88)
Hcat
2d-MS 1.7c-2 1.4c-1
4.4e-1
3.50-11-158.0%)
4.0c-1
3.8c-16:144.05))
8.7e-3(48516)
7.2e-2:47.2%)
2d-CG 1.9e22 2.4c-2
1.30-1 (-44019))11)))
6.1c-1
7.0c-1
2.0c-2(3:29)
2.1e-2(9.99)
2d-C
33.9--3 4.5e-2
3.9e-1
3.0e-11-565.09
2.8e-1
2.4e- 1(4227.99))
2.1e-3(4.665)
3.46-2(23.68)
NS
21-CG
1.2e-2 7.7c-2
2.Se-1
I. (611-19)1999))
1.0c+0
1.0e+0
1.2e-20089)
(.65-2(1.5%)
1d-C
[2.2e-1 4.8c-1
(.1e-1(47.7%)
4.0c-28118%
1.75-1(2.%)
2.1e-1(3.69)
4.4c-1(8.6%%
Wave
9.2c-1 2d-CG
1.0++0(..4)
1.3c+00
(2.0002990)))
2.2e-11:33.6%
1.3c+0)38.288
Chaotic GS |1.1e22 9.3c-
2.00-2(74.56)
9.40-2(-0.5)
8.6c-1
9.70-19957.89)
9192-3(13.29)
9.2e-2:1.8)
High- PNd 1.5e-2 5.3e-3
46.4--2 (3222)
3,3e-2/-5322222 532.99))
NaN
NaN
5.70-3160.56)
1.8--366.5%)
dim HNd 2.9e-2 1.1e-2
2.1e-3:92.99)
2.1e-3(51.4 445)
NaN
NaN
e.
8.3e-3
Proportion of improved tasks
12.5%
12.5%
12.5%
25.0%
81.3%
81.3%
1d-C 19.0e:3 1.4c-2
3.3e-1
4.8e-1
(.55-2(40099)
3.0e-1
9.0e-30228)
1.3e-2(8.699)
Burges
2d-C 4.44c-1 4.9c-1
4.9c-1c11008)
3.4c-11-10.59
1.3c+0.
13-+00-171.88)
4.3e-110.48)
1.92-110.2%)
2d-C 6.8c-1 6.7c-1
7.30-1(6.8%)
6.7e-1(-12%)
9.0e-10:3.999
9.4c-1(-005%)
7.24-1(5.19)
6.3c-1(5.2%)
6.3- 7.0e-1 1
7.8e-1(22.7%)
8.
6.0e- 165.68)
7.0e- (600689)
5.9e-1(6889
6.9e-1(1.28)
Poisson
4.2-G 5.0e-
4.58-167.8%
4.66-1(9.5%)
8. 1c-55.95
((-8-1(-8888)
4.1e-1(138)
5.1e-16005%)
(al-MS8..9-) 7,5e-1
5.1.-1643.5%)
4.7c-1(36.45)
9. 9.0.0.0.0
9.4c- (1-2579)
(.15-1
6.7e-1(10.5)
50
Cal-VC]1.ec+01.33
3.4c+1
2.70+1
1.244016.95
1.1e+011.5%)
1.2e+0ra1.19)
I.le+017.29)
FLS
Hcat
2d-MS 6.2e-2 4.8e-2
1.2c+0
9
1.0c+0
9.9c-1
(.4c-(t-33.399)
8.44-2).74.29)
2d-CG
1.5e-2 2.5-G
2.5e-2
9.9e-2 153.19))
1.2e-1
394.99))))
3.4e+0
4.4e+0
1.20-2.20.4%)
2.5e-2/0.6%)
2d-C 28.3e22 6.9e-2
44
14.9e (1-43.00
(1700))4
2.8e-
2.5e-
(6--16222.99)
4.8e- ((42244).48)
4.9e-2/9.58)
NS
2d-CG 1.2c-1
3
3.9e-1r
1.
2.7--1
9.9e-1-458..
1.
((1.88))
9.9e-2(18.0%)
1d-C |4,0c-1 4.1e-1 1elleel100
(65.5%) (5..79
1.1e-2/7722)) 1.1e-2/97.4%)
3.8c- (1.331))
3.9e-1449%)
Wave
2d-CG2.5c+02.4c-+
1.
1.9c+0x22.99)
2. 1..+019.1%)
2.0c+0(15.99)
1.7c+0(31.69)
1.7e++030.9%)
Chaotic GS 221ee22 9.4c-2
2.66-1
((15s
9.7c-1
1.0+00
2.2c-2,588)
9.0e-2(399))
High- PNd 8.9e-4 1.0e-3
3ee-33...-----11--00 (9) 2. ((-3
NaN
NaN
5.46-409.69)
6.4e-4(7.55)
dim
HNd 3.7e-3 4.0e-3 9 66-3(-160.88) 9. 96-36
NaN
NaN
1.3e-3(6388))
1.4e-3166.14
Proportion of improved tasks
12.5%
18.8%
25.0%
18.8%
81.3%
87.5%
31

---

Table 9: Full results of gPINN 155 <PINN 18 and RoPINN under different base models on PINNacle [2] (16 different PDEs). lower rMAE or rMSE with higher relative promotion indicates better performance. The promotion over vanilla is recorded in parentheses. For clarity, we highlight the value with blue if it surpasses the vanilla PINN, gray if it fails (over 10 times worse than the vanilla PINN), or is numerically unstable (NaN) or out-of-memory (OOM). For PINNerormer, fails in most of the tasks due to the OOM problem. We omit these tasks in calculating proportion.
Vanilla
gPINN
SS
<PINN [8
RoPINN (Ours)
(Part II)
PDE
iMAE
rMSE]
IMAE
iMSE
rMAE
iMSE
IMAE
MSE
Id-C |9.3e33 1.4c-2
6.5e-1
6.7e-1
5,5c-1
5,7c-1
8.00--3(139%)
1.0-2(26.5))
Burges
2d-C [OMM
0OM
0OM(-)
0OM(-))
00M(-5)
00M(-))
00M-5)
(OM(4)
2d-C
1.2e-1 6.6e-1
1.0++0039.448)
1.0e+0r-52.9%)
1.0e+0c59.48))
1.0++0(52.7%)
6,99-1(4119)
(.
2d-CG
5.4e-1 6.3e-1
1.0c+01-86.69)
1.0c+01-9945)
(.0++0(8666))
10e+06-59.95)
(.70-1(3.0%)
115 (S)
Poisson
PINNsformer S8
3d-CG
OOM 0OM
0OM-8)
0OM-8)
OOM(3)
OOM
(OM(
0OM:5)
24-MS
1.3e+0
le
OOM
OOM
00M
00M
(.24-1(42449)
(.5
2d-VCI
20M OOM
0OM
OOM--5
(0M-9)
00M-s)
O0M
00M:8)
(OM
Heat
2d-MS
OOM
0OM
0OM-5)
(OM-8)
00M(99)
00M:5)
(OMBR)
(OM:9)
2d-CG 0OM 0OM
(0M
(0M(9)
00M:%)
OOM
OOM:
(0M:
2d-C OOM
0OM
0OMes)
00Mc8)
00Mc9)
00M
0OM
(OM:
NS
2d-CG 1.0c-1 7.0e-2
(4-166660)
99)99.9)))
1.0++0(870099)
1.0c+0
9.8e-2/4.78)
(.3
1d-C
55.0c1
5.1c-1|
3.30-1166358,
8.5c-1165.5%)
5.20-1(3598
5.30-1:3.4%)
4.7e-1(7.4)
4.8e-1 (
Wave
2d-CG OOM
0OM
0OMe5)
0OMe8)
00M(3)
00M
0OM(R)
0OM:5)
Chaotic
GS
|OOM O0M|
0OMe5)
00Me5)
00M(5)
00M-s)
00Mc5)
0OM:55
ligh
PNd
OOM
0OM
0OMe8)
0OMe8)
00M
00M-8)
OOMeR)
0OM:9)
dim
HNd
0OM
00M
0OM %)
0OM(%)
00M %)
00M %)
OOMe5)
0OM:4) ())
Proportion of improved tasks
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
Id-C 2.76-1 5.4c-1
5.
5.66-1em..8)
6.70-1(22.09)
3.9c-
((47.58)
5.5c-1(-13%) NaN
(..
4.8e-1(12.59))
Burges
2d-C 8.77-1 9.4e-1
1.5e+0:71.88)
1.70+0(83.55)
NaN
(.36-1(4.666)
((24-1(2.48)
2d-C 17.3e-1 6.8e-1
7.4e-16114%)
(.8e-1(a.19)
1.6e-1.8.28)
1.8e-1(88449)
6.3e-1(3.4%)
6. Se-1(-0.15)
2d-CG 7.7e-1 8.1e-1
8.1c-1(5.5%
8.6c-1(66995
6.5e-1(149%)
7.38-199.8))
6.5e-1(15.9%)
6.9e-1(440%)
Poisson
3d-CG
7.5e-1 1.4e+0
2.0e+0(-163.68)
1.80+00-25559))
NaN
NaN
(44.71())75)
6.8e-1
2d-MS
9 9.8e-1
1.0c+065.08)
1.0c+06227%)
9.9e- ((-469)
1.0e+0g2.3%)
9.6e-1(2 9.34-1(3.4%)
(2.)
2d-VC|1.9es1e 1.5e++
3.3
2.7.8%
8.9c-1:9533%)
9.1c-199105)
6.0e+0068.7%) 4.66-0068.49
Heat
2d-MS 1.4c+0 1.1c+0
2.0.40)39.86)
1.4+00028.49))
8.3e-1(4288)
7.5c-1B144%
7.4e-1/4..95)
6.9e-1(3688)
2d-CG 5.0c-1 5.3c-1
8.20-1
7.30-1(37.22%)
8.1c-I ((4008)
9.0c- ((-e-1))925)
((1.-)))
5.0e-1(4.9)
2d-C |5.0c11 4.1c-1
8.
6.76-1(-6588)
(58.6)))
1. ((182118)88))
3.9c- ((2.9%
NS
2d.CG
9.9e-1
6.4e-1
5.4e- 1145.69)
4.1e- 113679))
1.0e00-.11))
1.0e+06-56119
(.67-1(5377)
1.4
Id-C
|4.7e-1
4.7e-1
8.68-1(85.2%)
8.8c-16-89.200
1.9--158.45) NaN
2.1e-1(55.8%)
4.6e-1(1.7)
4.6e- (0le1(119%)
Wave
2d-CG 1.740 1.1c#037.4%)
1.1e4033.4%)
NaN
9.8e-164328)
9.6e-1(41.19)
Chaotic
GS
NaN
NaN
Llee+0.9)
9.4c-1(-9)
8.5c-1es0
9.6c- ((-))
7.5e-1u%%
7.0e-I:-%)
High- PNd 4.6c-4 5.6c-4
288-3 518.45)
NaN
NaN
3.70-41999)
5.36-4(5.5%)
dim
HNd
1.9e-3 3.4e-3 5.9e-4(68.09)
8.0e-4/85 199
NaN
NaN
1.5e-3(203))
(8.
Proportion of improved tasks
31.3%
31.3%
43.8%
43.8%
100.0%
93.8%
F Related Work
This section will discuss some related works as a supplement to Section 2 We will first discuss some PINN research and then we will also clarify some looking similar but completely distinct topics.
PINN optimizers As we mentioned in the second paragraph of the introduction, many previous works focus on developing efficient and effective acep-mode) optimizers for PINNs [55] 37). which may help the optimization process tackle the ill-conditioned Hessian matrix naturally balance multiple loss terms [53]. As we formalized in Algorithm II] RoPINN is not restricted to a certain optimizer. The researchers can easily replace the Adam [1] L-BFGS (27 with other advanced optimizers. Since we mainly focus on the objective function, these works are orthogonal to us.
Numerical differentiation for objective functions In addition to the regularization or variational- based methods, some rescarchers attempt to replace the automatic differentiation with numerical
32

---

approximations [39 91. which can tackle the expensive computation cost caused by calculating high-order derivatives. However, this paradigm does not attempt to change the objection function definition, just focuses on the calculation of point optimization PINN loss, which is distinct from our proposed region optimization paradigm.
Sampling-basd based methods Strategies for sampling collocation points perform an important role in training PINN models [46]. Previous sampling-ber -aanedd-bdddddee methods mainly focus on accumulating collocation points to high-residual areas [51 23 or considering the temporal causality [45], whose theoretical analyses are usually based on the quadrature theorem [32]. Distinct from these methods, RoPINN is motivated by the optimization deficiency of PINN models and can be seamlessly integrated with sampling-
methods with significant promotion (Table 3). indicating that RoPINN works orthogonally to sampling methods. Besides, the theoretical analysis of RoPINN also starts from the optimization perspective, which reveals that one key advancement of RoPINN better balance between optimization eгror and generalization error (Theorem 3.12).
In addition, RoPINN is also distinct from data augmentation or adversarial training techniques in the following aspects: (1) Theorem difference: although our proposed practical algorithm is based on Monte Carlo sampling in a region. the underlying theoretical support and insights are region-based objective function (Theorem 3.5) (2) Implementation difference: In our algorithm, not only is the input changed, but the objective is also correspondingly changed. Thus, this paper is foundationally different from augmentation and adversarial training in that the ground truth label is fixed. Our design tailored to the physics-infon informed loss function of PINNs, where we can accurately calculate the equation residual at any point within the input domain.
G
Limitations
This paper presents region optimization new PINN training paradigm and provides both theo- retical analysis and practical algorithms, supported by extensive experiments. However, there are still several limitations. In the theoretical analysis, we assume that the canonical loss function is -LLipschiz---ssmooeee which may not be guaranteed in practice. Besides, RoPINN involves several hyperparameters, such as initial region size r. and number of past iterations To. Although we have studied the sensitivity w.r.t. them in Figures 2 and Appendix [.] and demonstrate that they are easy to tune most cases, we still need to adjust them for better performance in practice.
According to our experiments and theorems, we provide some recipes for hyperparameter tuning in the following, which may be helpful to the usage of RoPINN:
As shown in Figure region size will be progressively adjusted by RoPINN. Setting r in [10-6 10-1) can work well. According to Theorem 3.12 the choice of should balance optimization and generalization, which may be inherently decided by the PDE smoothness.
As analyzed in Figures 3 and 4 sampling 1-30 points can gain consistent promotion but will linearly increase the computation costs. Following our default setting (sampling one point) can already achieve a competitive performance
wide range of PDEs.
As presented in Figure 8 number of past iterations To is easy tune in 21. Setting To E {5, 10} can be a good choice, which has been widely verified in our paper.
Some hyperparameter tuning tools, such as Weights and Bias (Wandt)) p, may mitigate this limitation to some extent, which has already been used in previous related work (37]
H Broader Impacts
In this paper, we develop new region optimization training paradigm for PINNs and provide both theorem analyses and practical algorithms. This new perspective may inspire the subsequent research of PINNs, especially rethinking the canonical objective function. In addition, our proposed RoPINN shows favorable efficiency and generalizes well in different base models and PDEs, which can be used to boost the precision of PINNs and generally benefit the downstream tasks, such as physics phenomenon simulation. biological property analysis, etc. Since we purely focus on the training algorithm of PINNs, there are no potential negative social impacts or ethical risks.
https///gituub.co//wwa/t///
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper proposes a new training paradigm for PINNs along with a practical
algorithm. Theoretical analyses (Theorems 3.13..12 extensive experiments (Tables 2
a and detailed discussions of related work (Section and Appendix (F) are provided to verity the effectiveness of our design and elarify the scope of paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in Appendix G
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: We have provided complete and detailed proofs for all the lemmas, theorems and corollaries in Appendixes
and B The previous works have also been properly cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have clearly formalized our method in Algorithm I which is easy to reproduce. We have also included every detail of datasets, baselines and hyperparameters of our algorithm in both the main text and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
35

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is available at this repository: ststps://github..co///ii
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blccggudeeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included all the details in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have included the statistical significance in Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
36

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included these details in the main (Section and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have strictly followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included the discussion in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the data and code used with proper citation and license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "zQdG8idSKy", "reviewer": "Reviewer_aU22", "summary": "The paper proposes a novel optimization method for training physics-informed neural networks (PINNs): Region optimization, which extends a regular pointwise optimization of PINNs to neighborhood regions, named RoPINNs. The paper provides theoretical analysis explaining the decrease of generalization error with RoPINNs and high-order PDE constraints satisfaction. Then the paper presents a practical algorithm to enable the RoPINNs paradigm by introducing Monte-Carlo approximation of region integral and a region calibration scheme. Lastly, the paper assesses the performance on several well-known benchmark problems and showed the improved performance over the considered baselines.", "questions": "- Eq (5) seems to suggest that region optimization is applied to the boundary condition as well as L = L_bc + L_ic + L_pde. Is this the correct understanding or is it a typo?\n\n- The proposed optimization method seems to benefit significantly in the case of 1D reaction case while the benefit in 1D Wave or 1D convection cases are not as significant as that of 1D reaction. That is, rMSE for example in Table 2 of 1D reaction achieves an order (or orders) of magnitude improvement over the second best performing methods. Do the authors have some explanation on why?", "limitations": "- some more discussions on practical guidelines would be needed for users who want to utilize this method in different applications \n\n- some additional experiments (regarding computational wall time) would be needed to provide a complete picture of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and tackles the important problem in training PINNs (leveraging more information than just a point-to-point type mapping).\n\n- The paper presents theoretical analysis on benefits of RoPINNs, decreased generalization errors and satisfaction of higher-order PDE constraints.\n\n- The experimental results show that the proposed algorithm is effective in solving some challenging benchmark problems (known as failure modes) and is capable of producing more accurate solution approximates.", "weaknesses": "- Although shown to be very effective in several benchmark problems, the paper does not seem to provide general guidelines on how to set some important hyper-parameters such as initial region size and the number of sampling points. (While acknowledging that the authors indicate this as one of the limitations,) it would be great to see some experts’ guidelines.\n\n- If the authors could provide some analysis with regards to computational wall time, that would provide more complete pictures on how the proposed method performs. For example, it would be information to see a figure depicting a scatter plot showing computational wall time versus final rMSE type information, where a point in the plot corresponds to a different hyper-parameter setting (that is, the number of sample points). \n\n- [minor] there is a typo in the second paragraph of Section 4.2: line 289 Figure 2 => Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer aU22\n\nWe would like to sincerely thank Reviewer aU22 for providing a detailed review and insightful questions.\n\n> **Q1:** \"The paper does not seem to provide general guidelines on how to set some important hyper-parameters. It would be great to see some experts’ guidelines.\" \"More discussions on practical guidelines would be needed.\"\n\nMany thanks for reviewer's valuable suggestions.\n\n**At first, we would like to highlight that we adopt the same hyperparameter setting (initial region size $r=10^{-4}$, sample 1 point) for all benchmarks, across 19 different tasks and 5 different base models, which works well.** This verifies the consistent effectiveness of our algorithm.\n\nNext are some guidelines for RoPINN. We will include them in the revised paper.\n\n**(1) Experiment guidance: recap hyperparameter analyses in our paper.**\n\nWe have provided analyses on every hyperparameter of RoPINN in $\\underline{\\text{our original submission}}$, which deliver the following empirical guidance:\n\n- $\\underline{\\text{Figure 2 of Section 4.2}}$: Initial region size $r$ will be progressively adjusted during training.\n- $\\underline{\\text{Figure 3 of Section 4.2}}$: Increasing the number of sampling points will speed up the convergence but will bring more computation costs (see $\\underline{\\text{Q2}}$ for efficiency comparison).\n\n**(2) Theorem guidance: a new generalization bound for practical algorithm.**\n\nTo provide a more direct understanding of the practical algorithm, we prove a new theorem in the $\\underline{\\text{Figure 3 of Global Response PDF}}$, which considers the gradient estimation error:\n\n$\\text{Generalization\\ error}\\leq \\big((1-|\\Omega_r|/|\\Omega|)L+\\mathcal{E}_{r, \\mathrm{grad}}\\big)\\frac{2L}{|\\mathcal{S}|}\\sum\\alpha_t$\n\n$\\mathcal{E}_{r, \\mathrm{grad}}$ denotes the gradient estimation error. Here are the guidances derived from this new theorem:\n\n- Increasing the initial region size can benefit the first term of the bound, but may increase gradient estimation error. Fortunately, it can be adaptively adjusted by RoPINN during training, making this hyperparameter easier to tune.\n- Sampling more points can reduce the gradient estimation error.\n\n**(3) Practical suggestions.**\n\nBased on experiments and the new theorem, we suggest that\n\n- Firstly, researchers can use the same configuration as us, whose effectiveness has already been verified in 19 tasks and diverse base models. Specifically, as stated in $\\underline{\\text{Section 3.2 and 4 of main text}}$, initial region size $r=10^{-4}$ and number of sample points $=1$.\n- Further, they can adjust the initial region size according to the training loss curve. A jitter curve indicates that you should decrease the initial region size value. Besides, the number of sampling points should be set according to the efficiency demand, where more points will generally bring better results.\n- As discussed in $\\underline{\\text{Limitations in Appendix G}}$, some tools (e.g. Weights and Bias) can be helpful.\n\n> **Q2:** \"It would be information to see a figure depicting a scatter plot showing computation time versus rMSE type information.\" \"Experiments (computational wall time) would be needed to provide a complete picture of the proposed method.\"\n\nFirstly, we want to emphasize that **RoPINN is proposed in the spirit of boosting PINNs without extra backpropagation or gradient calculation (see $\\underline{\\text{Abstract}}$).** Thus, we only sample 1 point in all experiments, which has already achieved significant promotion and comparable efficiency w.r.t. PINNs.\n\nAs per the reviewer's request, we plot the performance and efficiency under different numbers of points in $\\underline{\\text{Figure 1 of Global Response PDF}}$. Here are the results of 1D-Reaction. 1D Wave is also included in the $\\underline{\\text{PDF}}$.\n\nWe can find that (1) computation costs will grow linearly when adding points; (2) more points will bring better performance, but will saturate around 10 points. The performance fluctuations of 9, 13, and 30 points are within three times the standard deviations (Appendix D.4).\n\n| Number of Points | RMSE  | 100 iters time | GPU Memory |\n| - | - | - | - |\n| PINN             | 0.981 | 18.47s         | 1.44GB     |\n| RoPINN 1 Point   | 0.095 | 20.04s         | 1.48GB     |\n| RoPINN 5 Points  | 0.050 | 46.48s         | 4.24GB     |\n| RoPINN 9 Points  | 0.033 | 67.98s         | 6.74GB     |\n| RoPINN 13 Points | 0.035 | 92.48s         | 9.24GB     |\n| RoPINN 30 Points | 0.037 | 196.41s        | 19.80GB    |\n\n> **Q3:** \"A typo in the second paragraph of Section 4.2: line 289 Figure 2 => Figure 3.\"\n\nMany thanks for your detailed review. We will correct this in the revised paper.\n\n> **Q4:** \"Eq (5) seems to suggest that region optimization is applied to the boundary condition as well as L = L_bc + L_ic + L_pde. Is this the correct understanding or is it a typo?\"\n\nYour understanding is correct. As described in $\\underline{\\text{Lines 113-115}}$, region optimization is applied to the inner domain, boundary and initial conditions. However, we will restrict all the sampled points within the definition domain of the PDE. We will include these details and rephrase Eq (5) in the revised paper.\n\n> **Q5:** \"The proposed method seems to benefit significantly in the case of 1D reaction while the benefit in 1D Wave or 1D convection cases are not as significant as that of 1D reaction.\"\n\nSince we experiment with RoPINN on diverse base models and various PDEs, **the value of relative promotion is affected by both base model capability and PDE difficulty,** which makes comparisons among different tasks a large variation.\n\nFor example, for the base model PINN, the promotion on 1D-Reaction is larger than 1D-Wave and Convection. In contrast, as for KAN, the promotions on 1D-Wave and Convection are larger than 1D-Reaction. Promotions for PINNsFormer on three tasks are close to each other. \n\nThus, comparing relative promotion values among different tasks may be meaningless.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0I4bArmEUb", "reviewer": "Reviewer_mhfV", "summary": "The authors developed a region optimized PINN to improve the prediction accuracy compared to the scatter-point based PINN.", "questions": "(1)  It is suggested to add some descriptions of training difficulty factors for the canonical PINN on 1D-Reaction in Section 4.2. \n(2)  Can the proposed method find a good number of sampling points well balancing the computational cost and convergence speed in Figure 3.\n(3)  The motivation of using Monte Carlo approximation should be elaborated. Why don’t the authors choose some other more advanced methods to provide better accuracy.\n(4)  The authors should add more details about the possible practical applications with the canonical loss function of L-Lipschitz-β-smooth.", "limitations": "Some initial guess methods can be developed to efficiently determine the preferable region size and sample number.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors proposed the region optimization paradigm and conducted a theoretical analysis.", "weaknesses": "The practical application scope is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer mhfV\n\nWe sincerely thank Reviewer mhfV for providing valuable feedback and suggestions in new experiments.\n\n> **Q1:** \"Add some descriptions of training difficulty factors for the canonical PINN on 1D-Reaction in Section 4.2.\"\n\nWe will add \"Previous research [22 of our paper] demonstrates that 1D-Reaction contains sharp transitions, which is hard to approximate\" into $\\underline{\\text{Section 4.2}}$.\n\n> **Q2:** \"Can the proposed method find a good number of sampling points well balancing the computation and convergence.\"\n\nNumber of sampling points is set manually. Our method can only adaptively adjust region size.\n\n**(1) The official design of RoPINN is sampling 1 point in each region, which is already a good and well-verified choice.**\n\nNote that RoPINN is proposed in the spirit of boosting PINNs **without extra backpropagation or gradient calculation (see $\\underline{\\text{Abstract}}$).** Thus, as described in $\\underline{\\text{Section 3.2 and Algorithm 1}}$, **we only sample 1 point in practice and adopt this as the official design.** \n\n$\\underline{\\text{Tables 2,3}}$ demonstrate that this official setting achieves significant promotion and comparable efficiency w.r.t. diverse PINNs in all 19 tasks. So, as for your question, we think sampling 1 point is already a good choice in balancing performance and efficiency.\n\n**(2) Discussion about sampling more points.** \n\nSince the convergence of deep models is affected by many factors (e.g. task, base model, optimizer), we cannot give a universal choice of sampling points. But we experiment RoPINN with more sampling points and plot an overview curve in $\\underline{\\text{Figure 1 of Global Response PDF}}$. Here are part of the results.\n\n| 1D-Reaction      | RMSE  | 100 iters time |\n| - | - | - |\n| Vanilla PINN     | 0.981 | 18.47s         |\n| RoPINN 1 Point   | 0.095 | 20.04s         |\n| RoPINN 5 Points  | 0.050 | 46.48s         |\n| RoPINN 9 Points  | 0.033 | 67.98s         |\n| RoPINN 13 Points | 0.035 | 92.48s         |\n| RoPINN 30 Points | 0.037 | 196.41s        |\n\nWe can observe that adding points will bring better performance but will saturate around 10 points. The performance fluctuations of 9, 13, 30 points are within three times the standard deviations (Appendix D.4).\n\nThus, according to our experiments, we first recommend sampling 1 point (our official design). If they want to obtain better performance, try [1, 10].\n\n> **Q3:** \"The motivation of using Monte Carlo. Why don’t the authors choose some other more advanced methods.\"\n\n**Again, RoPINN is proposed to boost PINNs without extra backpropagation or gradient calculation (see $\\underline{\\text{Abstract}}$).** Monte Carlo approximation can work well with 1 sampling point, which is easy to implement and without extra gradient calculation. As for other advanced methods, they usually need more points to achieve an accurate approximation. \n\nWe experiment with 2D space Gaussian quadrature, which requires square number points and the 1-point situation will degenerate to the center value. Here are the results of PINN+RoPINN.\n\n| 1D-Reaction rMSE | Monte Carlo | Gaussian Quadrature |\n| - | - | - |\n| 1 point   | **0.095**   | 0.109               |\n| 4 points  | 0.066       | **0.059**           |\n| 9 points  | 0.033       | **0.030**           |\n\nWe can find that under our official setting (sample 1 point), Monte Carlo is better. Although Gaussian quadrature is better in more points, these settings defeat our purpose in \"no extra gradient calculation\".\n\n> **Q4:** \"The practical application scope is limited.\" \"More details about the possible practical applications with the canonical loss function of L-Lipschitz-β-smooth.\"\n\n**(1) Theoretical assumption will not affect the practicability of algorithms.**\n\nA typical example is Adam, whose convergence can only be analyzed under strict assumptions, e.g. convex or L-Lipschitz-β-smooth [1]. However, it has been used as a foundation optimizer.\n\nIn our paper, the L-Lipschitz-β-smooth assumption is only in theoretical analysis for a basic understanding of our paradigm and will not affect the practicability of RoPINN.\n\n[1] Implicit Bias of AdamW: ℓ∞ Norm Constrained Optimization, ICML 2024\n\n**(2) RoPINN achieves consistent promotion for 5 diverse backbones, covering 19 tasks ($\\underline{\\text{Tables 2,3,7,8}}$).**\n\nOur experiments are much more extensive than the latest papers, such as PINNsFormer (ICLR 2024) and KAN (arXiv 2024). As described in $\\underline{\\text{Appendix C.1}}$, we experiment with diverse PDEs and do not constrain the loss function to be L-Lipschitz-β-smooth in practice. We believe that such extensive experiments strongly support the practicability of RoPINN.\n\n**(3) Our assumption is widely used in other theoretical papers.**\n\nThe papers that we cited in $\\underline{\\text{Theorem 3.3 of main text}}$ for point optimization analysis are based on L-Lipschitz-β-smooth assumption [13, 47 in our paper]. Other papers that we cited in $\\underline{\\text{Section 4 of main text}}$ also assume the smoothness of loss function [20 in our paper] or even \"inﬁnitely wide neural networks\" [42 in our paper].\n\n> **Q5:** Some initial guess methods can be developed to efficiently determine the region size and sample number.\n\n**As stated in $\\underline{\\text{Section 3.2 and 4 of main text}}$, we adopt the same hyperparameter setting (initial region size $r=10^{-4}$, sample 1 point) for all benchmarks, across 19 different tasks and 5 different base models, which works well consistently.** Thus, this configuration can be a good initial guess.\n\nBesides, we can obtain some empirical guidance from:\n\n- $\\underline{\\text{Figure 2}}$: Region size $r$ will be progressively adjusted by RoPINN. Setting $r$ in $[10^{-6}, 10^{-4}]$ can work well.\n- $\\underline{\\text{Figure 3}}$ and results in **Q2**: Sampling 1-30 points can gain consistent promotion.\n\nAs discussed in $\\underline{\\text{Appendix G}}$, some tools (e.g. Weights and Bias) can also be helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AQagYK96s9", "reviewer": "Reviewer_KCE6", "summary": "This paper extends the optimization process of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from this new paradigm, which is implemented by a straightforward but effective Monte Carlo sampling method. By calibrating the sampling process into trust regions, RoPINN finely balances sampling efficiency and generalization error. Experimentally, RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation.", "questions": "1. Which results in section 4 are for comparisons with the losses with high-order terms and variational methods?\n2. How tight is the generalization error bound?", "limitations": "The case of sampling more than one points in each region is not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of extending the optimization process of PINNs from isolated points to their continuous neighborhood regions is novel.\n2. Theoretical results on generalization error, convergence rate and estimation error of sampling are provided. \n3. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from the region optimization paradigm and associated theoretical results,\n4. RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation.", "weaknesses": "1. It is better to include the main proof idea of theoretical results in the main text.\n2. Although generalization error bound is provided, an intuitive explanation of the reason behind the success of region optimization is desirable. For example, when sampling one point in each region, why is the total loss decreased compared with point optimization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer KCE6\n\nMany thanks to Reviewer KCE6 for providing the insightful review and questions.\n\n> **Q1:** \"It is better to include the main proof idea of theoretical results in the main text.\"\n\nFollowing your suggestion, we will add the following descriptions into the main text as a brief proof for $\\underline{\\text{Theorem 3.5}}$:\n\n\"The proof is based on an optimization perspective [13 of our paper]. Firstly, based on the Lipschitz assumption, we can transform the generalization bound to $L$ times the expectation of parameter distance. Further, region optimization paradigm will bring more 'consistent' gradient optimization direction than point optimization at each iteration, thereby benefitting the generalization bound.\" \n\n> **Q2:** \"Although generalization error bound is provided, an intuitive explanation of the reason behind the success of region optimization is desirable. For example, when sampling one point in each region, why is the total loss decreased compared with point optimization?\"\n\nWe will clarify this question from both theoretical and optimization views.\n\n**(1) Theoretical view: generalization bound for practical algorithm.** \n\nJust as the reviewer mentioned, the generalization bound is based on an ideal assumption, that is, we can accurately obtain the loss of a region. However, in practice, as the reviewer mentioned, we have to use some approximation methods.\n\nTo provide a more direct understanding, we also prove the generalization bound for practical algorithm in the $\\underline{\\text{Figure 3 of Global Response PDF}}$. This new theorem is extended from $\\underline{\\text{Theorem 3.5 of main text}}$ by considering the gradient estimation error caused by Monte Carlo approximation.\n\nFor clarity, we present the convex version here. See the $\\underline{\\text{Global Response PDF}}$ for the non-convex version and proof.\n$$\n\\text{Generalization\\ error}\\leq ((1-|\\Omega_r|/|\\Omega|)L+\\mathcal{E}_{r, \\mathrm{grad}})\\frac{2L}{|\\mathcal{S}|}\\sum\\alpha_t\n$$\n\n\n$\\mathcal{E}_{r, \\mathrm{grad}}$ denotes the gradient estimation error caused by approximation methods, which is generally proportional to $|\\Omega_r|$. We have the following observations:\n\n- **Canonical point optimization corresponds to an extreme case:** $|\\Omega_r|=0$ and $\\mathcal{E}_{r, \\mathrm{grad}}=0$. In this case, the above equation degenerates to the bound that we proved in $\\underline{\\text{Theorem 3.3}}$.\n- RoPINN presents a trust region calibration algorithm, which can **finely balance benefits from region loss** (the first term $(1-\\frac{|\\Omega_r|}{|\\Omega|})$) **and gradient estimation error** (the second term $\\mathcal{E}_{r,\\mathrm{grad}}$), which leads to a lower generalization bound.\n\nThus, the success of RoPINN is from our trust region calibration, which can achieve a better balance between generalization bound and gradient estimation error. We also have visualized the balancing process (change of region size during training) in $\\underline{\\text{Figure 2 of main text}}$.\n\n**(2) Optimization view: A multi-iteration understanding of why sampling one point works.** \n\nAlthough we only sample one point at each iteration, $\\underline{\\text{Lemma 3.10 of main text}}$ shows that during training, gradients difference on different sampling points of $\\underline{\\text{successive iterations}}$ will be close to different sampling points at the $\\underline{\\text{same iteration}}$.\n\nThis means (informally) that if we think about the training process in a multi-iteration view, the optimization is getting closer to sampling multiple points during training, despite these sampled points being dispatched into multiple successive iterations.\n\n> **Q3:** \"Which results in section 4 are for comparisons with the losses with high-order terms and variational methods?\"\n\nAs stated in $\\underline{\\text{Line 232 of main text}}$, gPINN is with high-order regularization and vPINN is based on variational formulation. We have compared them in every experiment of $\\underline{\\text{Table 2 of main text}}$, which provides solid support for the advancement of RoPINN over them.\n\n> **Q4:** \"How tight is the generalization error bound?\"\n\nFirstly, we would like to thank the reviewer's great question, which makes us calibrate the proof of $\\underline{\\text{Theorem 3.5}}$ and obtain a tighter generalization bound by refining one-step derivation. \n\nSee $\\underline{\\text{Figure 2 of Global Response PDF}}$ for the complete refined theorem and proof. In short, the refined bound is:\n$$\n\\text{Generalization\\ error}\\leq (1-|\\Omega_r|/|\\Omega|)\\frac{2L^2}{|\\mathcal{S}|}\\sum\\alpha_t\n$$\n\nWe think this generalization bound is quite tight. All the derivations of the proof can be strictly equal in some cases. Furthermore, if we consider two extreme cases, this generalization bound works perfectly:\n\n- $\\Omega_r=0$, this bound degenerates to the bound of point optimization ($\\underline{\\text{Theorem 3.3 of main text}}$).\n- $\\Omega_r=\\Omega$ means directly optimizing the whole domain, which corresponds to the ideal loss function. In this case, our proved generalization bound is just equal to 0. Note that, as discussed in $\\underline{\\text{Q2}}$, this case will still face a large optimization error in practice.\n\nWe will also update $\\underline{\\text{Theorem 3.5}}$ as the refined version in the revised paper.\n\n> **Q5:** \"The case of sampling more than one point in each region is not discussed.\"\n\nActually, we have discussed sampling more points in $\\underline{\\text{Figure 3 of main text}}$ as one of the main analysis experiments. \n\nFurther, during rebuttal, we also provide more comprehensive results in the $\\underline{\\text{Figure 1 of Global Response PDF}}$. The key finding is that (1) computation costs will grow linearly when adding points; (2) more points will bring better performance but will saturate around 10 points.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jxYOsBEN4Q", "reviewer": "Reviewer_zNBp", "summary": "The preprint proposes to replace the collocation based PINN loss by a sum of local continuous integrals over regions around the collocation points. These continuous integrals are then again discretized using Monte Carlo integration with a single quadrature pint. The authors furthermore propose to adapt the region size during training using gradient statistics.", "questions": "I struggle to see why the proposed method should work theoretically. I acknowledge the adaptive nature of the regions for the sampling but struggle to see how this might help to accumulate integration points in regions of, e.g., high residual. A visualization that this, or something along these lines that explains why the proposed method works well, happens would be helpful. Furthermore, I am not convinced that the theoretical analysis presented is meaningful, as it analyzes the integrals over the region as continuous objects. Please comment or clarify misconception.", "limitations": "See questions", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors report good empirical performance on a number of benchmark problems.", "weaknesses": "The introduction of continuous integrals over regions around the collocation points that subsequentially are discretized by Monte Carlo integration again seems tautological. After all, the loss function in PINNs is already a Monte Carlo discretization of a continuous integral (over the whole computational domain). Furthermore, the analysis that the authors present for the modified loss in equation (5) should not be carried out with the continuous integral over the regions $\\Omega_r$ but with its Monte Carlo approximation. Otherwise, the comparison to the discretized PINN loss is unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer zNBp\n\nMany thanks to Reviewer zNBp for providing an insightful review and valuable suggestions.\n\n> **Clarify misconception.**\n>\n> \"After all, the loss function in PINNs is already a Monte Carlo discretization over the whole computational domain.\"\n\nFirstly, we want to highlight that our theorem **considers the training process (see proof in $\\underline{\\text{Appendix A}}$)**, which is distinct from previous quadrature-based theorems [Siddhartha Mishra et al., IMAJNA 2023] that only focus on the integral approximation and do not consider the collocation point change during training.\n\n**Thus, it is clearly one-sided to think about our method only from the integral approximation view.** Under the model training context, the reviewer described \"whole-domain-discretization PINN loss\" corresponds to \"(Global sampling) Point optimization\" paradigm (defined below), which is different from the \"point optimization\" in our paper.\n\n|                        | Paradigms                            | Implementation                                               |\n| - | - | - |\n| Our paper              | Region optimization                  | sample within regions around collocation points and **keep changing during training.** |\n| Our paper              | (Fixed) Point optimization           | sample over the whole domain **at the beginning, but fixed during training** (canonical PINN [33 of our paper]) |\n| **Reviewer mentioned** | (Global sampling) Point optimization | sample over the whole domain and **keep changing during training** (RAR [46 of our paper]) |\n\n> **Q1:** \"The introduction of continuous integrals over regions around the collocation points that subsequentially are discretized again seems tautological.\n>\n> \"The analysis should not be carried out with the continuous integral over the regions Ω𝑟 but with its Monte Carlo approximation. The comparison to the discretized PINN loss is unfair.\"\n\n**(1) Requested theorem: region optimization with MC approximation.**\n\nAs requested, we prove the generalization bound with MC approximation in $\\underline{\\text{Figure 3 of Global Response PDF}}$, which is extended from $\\underline{\\text{Theorem 3.5 of main text}}$ by incorporating gradient estimation error.\n\nFor clarity, we present the convex version here. See the $\\underline{\\text{PDF}}$ for the non-convex version.\n$$\n\\text{Generalization\\ error}\\leq ((1-|\\Omega_r|/|\\Omega|)L+\\mathcal{E}_{r, \\mathrm{grad}})\\frac{2L}{|\\mathcal{S}|}\\sum\\alpha_t\n$$\n\n$\\mathcal{E}_{r, \\mathrm{grad}}$ denotes the gradient estimation error caused by MC approximation.\n\nWe can find that the above two point paradigms can be unified in our region optimization formalization:\n\n- **(Fixed) point optimization** corresponds to an extreme case: $|\\Omega_r|=0$ and $\\mathcal{E}_{r, \\mathrm{grad}}=0$.\n- **(Global sampling) point optimization** is equivalent to another extreme case: $\\Omega_r=\\Omega$. Since the larger region is generally harder to approximate, this may cause a large $\\mathcal{E}_{r, \\mathrm{grad}}$.\n- **RoPINN** adopts the trust region calibration to adaptively adjust $|\\Omega_r|$ during training.\n\n**Thus, introducing \"region\" is not tautological**, which\n\n- Provides a general theoretical framework for three paradigms.\n- Reveals the balance of generalization bound and optimization error.\n- Motivates RoPINN as a practical algorithm for balancing.\n\n**(2) All the experiments are fair.**\n\nSince we only sample 1 point in practice, RoPINN will not bring extra gradient calculation than \"discretized PINN loss\". **Under a comparable efficiency, RoPINN consistently boosts 5 different PINN models in 19 tasks ($\\underline{\\text{Tables 2,3,7,8}}$)**, which should not be overlooked.\n\n> **Q2:** \"Why proposed method should work theoretically. I acknowledge the adaptive regions but struggle to see how this help to accumulate points in high residual.\"\n\nRoPINN is not a collocation-point sampling algorithm, whose contribution is orthogonal to RAR [46 of our paper] (see $\\underline{\\text{Table 3 of main text}}$). Next, we will show why RoPINN works well.\n\n**(1) Theoretical understanding.**\n\nAs discussed in **Q1-(1)**, the generalization bound has two parts, the first term $(1-\\frac{|\\Omega_r|}{|\\Omega|})$ is inversely proportional to $|\\Omega_r|$ and the second term $\\mathcal{E}_{r, \\mathrm{grad}}$ is generally proportional to $|\\Omega_r|$.\n\n- (Fixed) and (Global sampling) point optimization are just two special cases. Obviously, they **lack flexibility and cannot adaptively balance optimization and generalization**.\n- RoPINN presents an adaptive algorithm, which can **adjust the region size for a better balance of the above two terms**. A visualization of the balance process is in $\\underline{\\text{Figure 2 of main text}}$.\n\n**(2) Experiment statistics.**\n\nWe also record the standard deviation of parameter gradients on the last 100 iterations, which reflects the training stability. Fixed point optimization leads to very stable training and Global Samlping brings more perturbations, while RoPINN achieves a relatively balanced value.\n\n| 1D-Reaction     | Paradigm                | rMSE      | Gradient Std           |\n| - | - | - | - |\n| **PINN+RoPINN** | **Region**              | **0.095** | 0.310                  |\n| PINN Vanilla    | (Fixed) Point           | 0.981     | 1.262$\\times 10^{-10}$ |\n| PINN+RAR        | (Global Samlping) Point | 0.981     | 1.128                  |\n\n> **Q3:** \"I am not convinced that the theoretical analysis presented is meaningful, as it analyzes the integrals over the region as continuous objects.\"\n\nFollowing your suggestion, we have extended our theorem to practical algorithms (see $\\underline{\\text{Q1-(1)}}$). Our original theorem analyses are under a perfect approximation, that is $\\mathcal{E}_{r, \\mathrm{grad}}=0$, which is still meaningful in:\n\n- Providing a unified framework for three paradigms.\n- Considerating the optimization process, thereby easily to be extended to practical algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response and Summary of Revisions

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

This paper proposes and theoretically studies a **new training paradigm as region optimization**, which can benefit both generalization bound and hidden high-order constraints of PDEs. RoPINN is derived from the theory as a practical training algorithm, which **consistently boosts the performance of 5 different PINN models on 19 PDEs without extra backpropagation or gradient calculation**. Detailed visualizations and theoretical analysis are provided.

The reviewers generally held positive opinions of our paper, in that the proposed method is "**novel**", "**practical**", "**well-motivated**", "**tackles an important problem**", and "**effective in solving some challenging problems**"; we have experimented "**on a wide range of PDEs**" and shown "**good empirical performance**".

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing detailed clarification, requested results and theorems. Here is the summary of the major revisions:

- **Provide practical guidelines for initial region size, number of sampling points (Reviewer aU22, mhfV):** Firstly, we highlight that our experiments are all under the same hyperparameter setting, which can be a good and widely-verified choice for initial guess. Secondly, we recall the hyperparameter analysis experiments and newly proved theorem to show both empirical and theoretical guidance. The requested experiment on more sampling points is provided for an overview of RoPINN.
- **Experiment with more advanced approximation methods (Reviewer mhfV):** Firstly, we emphasize the design principle of RoPINN: do not bring extra gradient calculation, which motivates us to choose the easy-to-implement Monte Carlo method. Secondly, we provide experiments on Gaussian quadrature, which needs more sampling points for an accurate approximation.
- **Explain how tight the generalization bound is (Reviewer KCE6):** Following the reviewer's question, we calibrate the proof of our theorem and obtain a tighter bound by only updating one-step derivation. The refined theorem is quite tight and can seamlessly cover two extreme cases. We will include the refined theorem in the revised paper.
- **Explain why our method works theoretically (Reviewer KCE6, zNBp):** Following the reviewer's request, we prove a new theorem for practical algorithm. By considering the gradient estimation error caused by MC approximation, we demonstrate that point optimization is just a special case of our theorem, which lacks flexibility. In contrast, RoPINN can adaptively balance generalization and optimization errors.
- **Explain the meaning of our region optimization theorem (Reviewer zNBp):** Firstly, we clarify that our theorem considers the training process, which may result in some misconceptions. Secondly, we show that our region optimization theorem provides a unified theoretical framework for different optimization paradigms and can be easily extended to practical algorithms.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. All the above revisions will be included in the final paper. We'd be very happy to answer any further questions.

Looking forward to the reviewer's feedback.

### **The mentioned materials are included in the following PDF file.**

- **Figure 1 (Reviewer aU22, mhfV)**: Efficiency & performance w.r.t. number of sampling points.
- **Figure 2 (Reviewer KCE6, zNBp)**: Refined Theorem 3.5: A tighter generalization bound.
- **Figure 3 (Reviewer KCE6, zNBp)**: New Theorem: Generalization bound under Monte Carlo approximation.

Due to the compilation difficulty, we can only provide a brief description of new theorems in the OpenReview reply. A formal version is in the PDF.

Meta-review: The submission has received mixed ratings after extensive post-rebuttal discussion between the authors and the referees. One of the referees recommends rejection based mainly on presentation aspects. The other three referees recommend acceptance, and they also rate the presentation as sufficiently clear. The authors are advised to take into account the detailed reviews as well as the clarifications provided in their rebuttal while revising the submission.
Decision: Accept (poster)