PINNACLE: PINN Adaptive ColLocation and Experimental points selection
Authors: Gregory Kang Ruey Lau
Apivich Hemachandra
See-Kiong Ng
Bryan Kian Hsiang Low
Venue: ICLR 2024 spotlight (2024)
Keywords: Physics-informed Neural Networks
PINNs
adaptive training points selection
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: A novel PINN training algorithm, motivated by analysis of the Neural Tangent Kernel, that jointly selects all training point types in the composite loss function to gain large performance boosts for forward, inverse, and transfer learning problems.

Abstract: Physics-Informed Neural Networks (PINNs), which incorporate PDEs as soft constraints, train with a composite loss function that contains multiple training point types: different types of collocation points chosen during training to enforce each PDE and initial/boundary conditions, and experimental points which are usually costly to obtain via experiments or simulations. Training PINNs using this loss function is challenging as it typically requires selecting large numbers of points of different types, each with different training dynamics. Unlike past works that focused on the selection of either collocation or experimental points, this work introduces PINN Adaptive ColLocation and Experimental points selection (PINNACLE), the first algorithm that jointly optimizes the selection of all training point types, while automatically adjusting the proportion of collocation point types as training progresses. PINNACLE uses information on the interactions among training point types, which had not been considered before, based on an analysis of PINN training dynamics via the Neural Tangent Kernel (NTK). We theoretically show that the criterion used by PINNACLE is related to the PINN generalization error, and empirically demonstrate that PINNACLE is able to outperform existing point selection methods for forward, inverse, and transfer learning problems.

Published as a conference paper at ICLR 2024
PINNACLE: PINN ADAPTIVE COLLOCATION AND EXPERIMENTAL POINTS SELECTION
Gregory Kang Ruey Lau Apivich Hemachandra"h
'Department of Computer Science, National University of Singapore, Singapore 117417 *CNRS@CREATE, Create Way, #08-01 Create Tower, Singapore 138602
(greglau, seevvcchh@ooog nus edu.sg sg
See-Kiong Ng Bryan Kian Hsiang Low
Department of Computer Science, National University of Singapore, Singapore 117417 seek seekiongenus. www.uuuuuuuuudduuueo.ommuuue.... sg, lowkh& nus .edd.edusss
ABSTRACT
Physics-Infone Informed Neural Networks (PINNs), which incorporate PDEs as soft con- straints, train with a composite loss function that contains multiple training point types: different types of collocation points chosen during training to enforce each PDE and initial/houndary conditions, and experimental points which are usually costly to obtain via experiments simulations. Training PINNs using this loss function is challenging it typically requires selecting large numbers of points different types, each with different training dynamics. Unlike past works that focused on the selection of either collocation experimental points, this work introduces PINN ADAPTIVE COLLOCATION AND EXPERIMENTAL POINTS SE- LECTION (PINNACLE), the first algorithm that jointly optimizes the selection of all training point types, while automatically adjusting the proportion of collo- cation point types training progresses. PINNACLE interaction among training point types, which had not been considered before, based on an analysis of PINN training dynamics via the Neural Tangent Kernel (NTK). We theoretically show that the criterion used by PINNACLE is related the PINN generalization error, and empirically demonstrate that PINNACLE is able to outperform existing point selection methods for forward, inverse, and transfer learning problems.
information the
INTRODUCTION
Deep learning (DL) successes in domains with massive datasets have led to questions on whether also be efficiently applied the scientific domains. In these settings, while training data may be more limited, domain knowledge could compensate by serving as inductive biases for DL training. Such knowledge can take the form of governing Partial Differential Equations (PDEs), which can describe phenomena such as conservation laws or dynamic system evolution in areas such as fluid Chen
dynamics (Cai et 2021 20en et al. 2021 Jagtap et al., 2022). wave propagation and optics (bin
Waheed et al. 2021 Lin & Chen 2022 or epidemiology (Rodriguez et al. 2023).
Physics-Informed Neural Networks (PINNs) are neural networks that incorporate PDEs and their initial/boundary conditions (IC/BCs) as soft constraints during training (Raissi et al. 2019). and have been successfully applied various problems. These include forward problems (i.e., predicting PDE solutions given specified PDEs and ICs/BCs) and inverse problems (i.e., learning unknown PDE parameters given experimental data). However, the training of PINNs challenging.
To learn solutions that respect the PDE and IC/BC constraints, PINNs need composite loss function that requires multiple training point types: different types of collocation points (CL points) chosen during training enforce each PDE and IC/BCs, and experimental points (ExP points) obtained via experiments or simulations that are queries for ground truth solution values. These points have
Equull contribution.

---

Published as a conference paper at ICLR 2024
different training dynamics, making training difficult especially in problems with complex structure. In practice, getting accurate results from PINNs also require large numbers of CL and Exp points, both which lead to high costs the former leads to high computational costs during the training process (Cho et al. 2023 Chiu et al. 2022), while the latter is generally costly to acquire experi- mentally or simulate. Past works have tried to separately address these individually by considering
an adaptive selection of CL points (Nabian et al.
2021 Gao & Wangl [023] Wu et al., [2023] Peng
al. 2022; Zeng et al. [022] Tang al. 2023 or EXP points selection through traditional active
learning methods (Jiang et al. 2 et a.. 2022). Some works have also proposed heuristics
that adjust loss term weights of the various point types to try improve training dynamics, but do not
consider point selection (Wang et al. (202cc). However, no work thus far has looked into optimizing all training point types jointly to significantly boost PINN performance.
Given that the solution spaces of the PDE, IC/BC and underlying output function are tightly coupled, it inefficient and sub-optimal to select each type of training points separately and ignore cross information across point types. For different PINNs or even during different stages of training, some types of training points may also be less important than others.
In this paper, we introduce the algorithm PINN ADAPTIVE COLLOCATION AND EXPERIMENTAL DATA SELECTION (PINNACLE that is the first to jointly optimize the selection of all types of training points (e.g., PDE and IC/BC CL and ExP points) given budget, making use of cross infor- mation among the various types of points that past works had not considered to provide significant PINNs training improvements. Our contributions
summarized as follows:
We introduce the problem of joint point selection for improving PINNs training (Sec. and propose novel representation for the augmented input space for PINNs that enables all types of training points for PINNs to be analyzed simultaneously (Sec. 4.1
With this augmented input space, we analyze the PINNs training dynamics using the com- bined NTK eigenspectrum which naturally incorporates the cross information among the various point types encoded by the PDEs and IC/BC constraints (Sec. 4.2
Based on this analysis, define new notion of convergence for PINNs (convergence degree) (Sec. (.3) that characterizes how much a candidate set involving multiple types of training points would help in the training convergence for the entire augmented space. We also theoretically show that selecting training points that maximizes the convergence degree leads to lower generalization error bound for PINNs (Sec. 4.3)
We present a computation method of the convergence degree using Nystrom approximation (Sec. [.1). and two variants of PINNACLE based maximizing the convergence degree while also considering the cvolution of the empirical NTK (cNTK) of the PINN (Sec. (.2).
We empirically illustrate how PINNACLE's automated point selection across all point types are interpretable and similar to heuristics of past works, and demonstrate how PIN- NACLE outperform benchmarks for range of PDEs in various problem settings: forward problems, inverse problems, and transfer learning of PINNs for perturbed ICs (Sec. 6
2 BACKGROUND
Physics-Informed Neural Networks. Consider partial differential equations (PDEs) of the form
Nu,, (x) = f(x Vr E X
and
[=(u((())) = gx(x)) Var E ox;
(1)
where u(x) is the function of interest over a coordinate variable defined on a bounded domain X' C Rd (where time could be a subcomponent,. /V is a PDE operator acting on u(x) with parameter:D 3, and B, [u] are initial/boundary conditions (IC/BCs) for boundaries 0x, C. X. For PINN training, assume operators A and B, and functions ff and g are known, while 3 may or may not be known.
Physics-Informet Neural Networks (PINNs) are neural networks (NN) ûg that approximates u(x)). They are trained on a dataset X that can be partitioned into X,, Xp and X2, corresponding to the
'To simplify notation, we write NV (8] as M[],, keeping the 3 dependence implicit, except for cases where the task is to leam B. Examples of PDEs, specifically those in our experiments, can be found in Appendix I.1
2

---

Published as a conference paper at ICLR 2024
ExP points, PDE CL points and IC/BC CL points respectively. The PDE and IC/BCs are added as regularization (typically mean saaannd errors terms) in the loss function
(ūo(x) x(x)))
(M(ūo)(z) - (x)))
(Blūa] g(x)))
[(ūo; X)
Σ DEX,
A
2N
2Np
2N,
DEX,
zEX,
(2)
where the components penalize the failure of üg in satisfying ground truth labels u(z), the PDE, and the IC/BCs constraints respectively, and X and A are positive scalar weights.
NTK of PINNs. The Neural Tangent Kernel (NTK) of PINNS can be expressed block matrix broken down into the various loss components in (2) (Wang et al., (02(a).. To illustrate, a PINN trained with just the ExP points X, and PDE CL points X, using gradient descent (GD) with learning rate 7 has training dynamics that can be described by
[ u0-- (X.) úg. (X.) [N (Xp)
10,00 2r,op Ot.ps
úo. (X.) ) -(())) [N f(xp)
-N Wiuo.1 (xpp)((p))
-1
@r.ps
Qt.pp
(3)
where Jt,s V Voûo, (X,), Jr.p VoN [ (X),), and Or,ss Je Qt.pp Jr Tpp and Or.op = O = Jt.. are the submatrices of the PINN empirical NTK (eNTK). Past works have analyzed PINNs training dynamics using the eNTK (Wang et al.) 202 la; 2022c Gao et al. 2023
We provide additional background information on NTKs for general NNs in Appendix D
3 POINT SELECTION PROBLEM SETUP
The goal of PINN training is to minimize the composite loss function separate loss terms corresponding to EXP (X, X), PDE CL (Xp C X), and multiple IC/BC CL (X, C ax) point&] Instead of assuming that the training set is fixed and available without cost, we consider a more realistic problem setting where we have a limited training points budget. The choice of training points then becomes important for good training performance.
2 which comprises of
Hence, the problem is to select training sets X,, Xp. and X,, given a fixed training budget, to achieve the best PINN performance") Due to high acquisition cost (e.g.. limited budget for conducting experiments), we consider fixed ExP training budget Xxl. We also consider combined training budget for the various CL point types Xxp + X k. which is limited due to computational cost training. Note that the algorithm allowed freely allocate the CL budget among the PDE and various IC/BC point types during training. in contrast to other PINNs algorithms where the user needs to manually fix the number of training points for each type. Also, while the ExP and CL points do not share a common budget, they can still be jointly optimized for better performance (i.e., EXP points selection could still benefit from information from CL points and vice versa).
4 NTK SPECTRUM AND NN CONVERGENCE IN THE AUGMENTED SPACE
4.1 AUGMENTED INPUT REPRESENTATION FOR PINNS
To analyze the interplay among the training dynamics of different point types, define new augmented space Z C X' x (8, p, b). containing training points of all types, as
2 (x, E X) U p( X) U (x, 0X}
where p and b are the indicators for the EXP points, PDE CL points and IC/BC CL. points respectively, to specify which type of training point is associated with. Note that a PDE CL point (z. p) E Z is distinct from an EXP point
(z, s), even though both points have the same
Forr notational simplicity we will denote a single IC/BC 1 in subsequent equations, however the results can casily be generalized to include multiple ICs/BCs as well.
TTo compare with the active learning problem setting, ExP points be viewed queries the ground truth oracle, and CL points as queries to oracles that advise whether ûg satisfies the PDE and IC/BC constraints.
'The specific metric will depend the specific setting: for forward problems it will be the overall loss for the function of interest uzx), while for inverse problems it will be for the parameters of interest
3

---

Published as a conference paper at ICLR 2024
coordinate We abuse notation by defining any function h X R to also be defined on 2 by h(2) h(x,1) & h(x) for all indicators {s, p, b}. and also define a general prediction operator F that applies the appropriate operator depending on the indicator r, i.c.,
(1))(z,s) h(x), F(N((r,)) p) [=)((r), and F1h](z, b) (l))
(5)
The residual of the network output at any training point 2 can then be expressed as Ro, (z) F Flūo,](2) Flu](z), and we can express the loss ( given appropriately set X and As. as
=
1 Σ Ro. (2) (2) 2
1 Σ ZEZ
L(uo; Z)
(F] -Flu(=)))
(6)
2
ZEZ
Using the notations introduced above, the goal of our point selection algorithm would be to select a training set S from Z which will be the most beneficial for PINNs training. As the point selection is done on Z space, all types of training points can now be considered together, meaning the algorithm could automatically consider the cross information between each type of points and prioritize the budgets accordingly (explored further in Sec. 6.2
4.2 NTK EIGENBASTS AND TRAINING DYNAMICS INTUITION
With this PINN augmented space, we can now consider different basis that naturally encode cross information the interactions among the various point types during training. For instance, the off-diagonal term St.sp in (3) encodes how the PDE residuals, through changes in NN parameters during GD, change the residuals of úg. This is useful as in practice, we usually have limited ExP data on úe but are able to choose PDE CL points more liberally. Past works studying PINN eNTKs Wang et al. 2021a 2022c) had only used the diagonal eNTK blocks (0),ss+ Ot.pp) in their methods.
Specifically, we define the eNTK in the augmented space as (2, = VLo,(e, VoF(ūe,]2")) Since O: is continuous, positive semi-definite kernel, by Mercer's theorem (Schölkopf & Smola 2002) we can decompose O, to its eigenfunctions vhe and corresponding eigenvalues Ar,a 2 0 (which are formally defined in Appendix E as
CORTS
Or(z, 212,z2)= = Σ Aes (L.())
03 13
Notice that since the eigenfunctions depend on the entire eNTK matrix, including its off-diagonal terms, they naturally capture the cross information between each type of training point. Empirically, we find that the PINN eNTK eigenvalues falls quickly, allowing the dominant eigenfunctions (i.e., those with the largest eigenvalues) effectively form a ba- sis for the NN output in augmented space. This is consistent with results from past eNTK works for shows how the PINN pre- diction values F( (2) in (z, 8) and (z, p) subspaces can be reconstructed by just the top dominant eNTK eigenfunctions. Further discussion and more eNTK eigenfunction plots are in Appendix [F)
Figure 1: Reconstruction of the PINN pre- diction values F2)))(z), after 10k GD train- ing steps, in (æ,s) and (x, p) subspaces of the ID Advection equation, using the dom- inant eNTK eigenfunctions (i.e., those with the largest eigenvalues).
vanilla NNs (Kopitkov & Indelman, 2020 Vakili et al.] (02T). Figure |
The change in residual at point E Z, given training set Z for GD step with small enough 17, is
A AR.. (z; 2) A Ro (z; 2) - Ro. (zi 2) R 11 des (Wr,lZ), Ro. (Z) Ho vt,(())
(8)
(x,((2)
Note that A Ro, (z;Z) describes how the residual at any point z (of any point type, and including points outside 2) influenced by GD training on any training set Z that can consist of multiple point types. Hence, ARe (z; 2) describes the overall training dynamics for all point types.
Foor compactness, we will overload the notations with Ro. (7) and O, (Z, 7') for cases of multiple inputs, and use (2) to refer to e, 2).
4

---

Published as a conference paper at ICLR 2024
(a) After 10k steps of GD
(b) After 100k steps of GD
spuder
500
Top 50 eg
3212
Reconstruction of the true PDE solution of the 1D Advection equation using eigenfunc- tions of θ at different GD timesteps. For each time step, we plot the eigenfunction component both for (I, s) subspace (top row) and (I, p) subspace (bottom row).
Figure 2:
We can gain useful insights on PINN training dynamics by considering the augmented eNTK eigenbasis. First, we show that the residual components that align to the more dominant eigen- functions will decay faster. During training period where the eNTK remains constant, i.c., Or(2,2)) = Ool2, 2¹), the PINN residuals will evolve according to ttta (Z) (Z)
Re (z; 2) Reo (z) Oo(z, Z)
We provide a proof sketch of (9) in Appendix From 9). we can see that the residual component that is aligned to the eigenfunction u (given by a,,s (Z)) will decay as training time progresses at a rate proportional to i) For training set points Z. the decay is exponential as ( further re-
20 duces to Di= e en)AAo. or.i (Z) , (Z). Interestingly, note that (9) applies to all 2, suggesting that choosing training point set Z with residual components more aligned to dominant eigenfunc-
tions also results in lower residuals for the entire augmented space and faster overall convergence.
Second, during PINN training, the eNTK eigenfunctions will gradually align to the true target so- lution. While many theoretical analysis of NTKs assumes fixed kernel during training, this does not hold in practice (Kopitkov& & Indelman 2020). Figure 2 Jshows that the dominant eigenfunctions can better reconstruct the true PDE solution training progresses, indicating that the eNTK evolves during training to further align to the target solution. Notice how the predictions on both subspaces needed be considered for PINNs, which had not been explored before in past works. While the top 10 eigenfunctions replicated the (I, p)-subpaace prediction better, more eigenfunctions were needed reconstruct the full augmented space prediction (both (z, and (I, p) subspaces).
This suggests that PINN training consists of two concurrent processes of 1) information propagation from the residuals to the network parameters mainly along the direction of the dominant eNTK eigenfunctions, and 2) eNTK eigenfunction evolution to better represent the true PDE solution. In Sec. 5.2 we propose point selection algorithm that accounts for both processes by selecting points that would cause the largest residual decrease to speed up Process 1, and re-calibrate the selected training points by re-computing the eNTK to accommodate Process
4.3 CONVERGENCE DEGREE CRITERION AND RELATIONS TO GENERALIZATION BOUNDS
The change in residual A Ro, (z; 2) 8 only captures the convergence at single test point While is possible to consider the average change in residual w.r.t. some validation set, the quality of this criterion would depend on the chosen set. Instead, to capture the convergence of the whole domain, we consider the function A Ro. (:: 2) in the reproducing kernel Hilbert Space (RKHS) He, of O,, with its inner product (,, ) He, defined such that (1a,,, th.j) He, = 8ig / Ignoring factors of n, define the convergence degree af to be the RKHS norm of ARe. (- Z))
azz) A ||ARo (: :2)1 Ho, = E A (-; ; 2), 2... = Σ (Z),
(10)
Selecting training points that maximizes the convergence degree could speed up training conver- gence, which corresponds to Process 1 of the PINN training process. Hence, our point selection algorithm will be centered around choosing a training set Z that maximizes a(Z).
5

---

Published as a conference paper at ICLR 2024
Relations to Generalization Bounds. We now theoretically motivate our criterion further by showing that training points that maximizes (10) will lead to lower generalization bounds.
Theorem (Generalization Bound, Informal Version of Theorem 2.. Consider PDE of the form 11. Let 2 = [0, 11dh x {s.P), and S C 2 be a i..d. sample of size Ng from a distribution Dg. Let úgg be NN which is trained on S by GD, with small enough learning rate, until convergence. Then, there exists constants C1, C22 C3 = (ppoy((1 / Amax (n(()) /mmin//mnnnn (O(())) such that with high probability over the random model initialization,
Eg-x 0و0 (x) u(z)]]] & Re (S) + (S) 1/2 + c3.
(11)
Theorem T is proven in Appendix G Our derivation extends the generalization bounds presented by Shu et al. (2022b), and takes into account the PINN architecture (which is captured by the eNTK) and the gradient-base based training method, unlike past works (Mishra & Molinaro] [021]] De Ryck & Mishra 2022). The term Re_ [ (S) is the training error at convergence of the training set, and can be arbitrarily small (Gao et al. 2023) which we also observe empirically. This leaves the generalization enror to roughly be minimized when a (Z) is maximized. This suggests that training points that maximizes the convergence degree should also lead to lower generalization error.
PROPOSED METHOD
5.1
NYSTROM APPROXIMATION CONVERGENCE DEGREE
In this section, we propose a practical point selection algorithm based on the criterion presented in During point selection, rather than considering the whole domain Z, we sample finite pool paol C E to act candidate points for our selection process. In practice, we are unable to compute the eigenfunctions of the eNTK exactly. Hence, we estimate the eNTK eigenfunctions and corre- sponding eigenvalues Vr..i and Aτ, using the Nystrom approximation (Williams & Seeger, 2000) Schölkopf & Smola (2002). Specifically, given a reference set Ziet of size p, and the eigenvectors it, and eigenvalues Xe, of e₂ (Zrrt,,, for each i I..... p. we can approximate thi, and Xe,5 by
T0P.
V2,2 = (z) A ViZratl 2 Oz(z, Ur, and de,t deS & |Zref] "x,,.
(12)
In practice, selecting p < Zpool is sufficient since NTK eigenvalues typically decay rapidly Lee et al. 2019 Wang et al. 2022c Given Li,s and Se,5 can approximate (Z) 8 as
(Z) ~ (Z) _ (2), Ro, (2)) = Zizz X.ب TT, 12 R) (2)
(13)
and subsequently approximate the convergence degree [00 as
(l R de (7)
Ae. 0z) = Σ X (1T 2) Ro, ()))
(14)
In Proposition]] 1 we show that the approximate convergence degree estimate for the true convergence degree.
[4] provides sufficiently good
Proposition 1 (Criterion Approximation Bound, Informal Version of Proposition D). Consider a and â defined in (10) and (14) respectively. Let Npval be the size of set Zpant Then, there exists a sufficiently large Puin such that i set Zeet of size p 2 Puin is sampled uniformly randomly from Zpoot. then with high probability, for any Z C Zpoot, it is the case that
(a(Z) (Z)l S ((Nraa/)) (Z)113)
(15)
Proposition I is formally stated and proven in Appendix H using Nystrom approximation bounds.
5.2 PINNACLE ALGORITHM
Given the -heory-inspired criterion rithm The algorithm consists of two alternating phases: the point selection phase, and the training phase. In the point selection phase, we generate a random pool of candidate points Zpumal (Line 4 then aim select subset Z C Zpaul which fits our constraints and maximizes à(Z) (L. How- ever, this optimization problem is difficult to solve exactly. Instead, we propose two approaches to approximate the optimal set for á (additional details are Appendix I.I).
we now introduce the PINNACLE algorithm (Algo-

---

Published as a conference paper at ICLR 2024
SAMPLING (S). We sample batch of points from Zpoot where the probability of selecting point Zooo is proportional to á(z). This method will select points that individually have high convergence degree, while promoting some diversity based on the randomization process.
Algorithm 1 PINNACLE
1: Input: PINN üg learning rate v, number of itera- tions T, eNTK appproximation error
2: repeat
3: // Point selection phase
4: Randomly sample candidates Zpoot from 2
K-MEANS++ (K). We represent each point with a vector embedding 2 + (24,2 az (()))== and perform K- Means++ initialization on these embedded
5: Compute + using Nystrom approximation
6: Select subset Z Zpool to fit constraint using SAMPLING or K K-MEANS++
7:
|/ Training phase
8:
Compute θ = O (Z poo))
vectors to select a batch of points. Similar
9:
for L' 1, 1 do
to previous batch-mode active learning algorithms (Ash et al. 2020), this method
10:
Op41
+ 02 NiVaL(un, 2)
select points with high convergence
11:
Exit training if ||@ Op (Zpoo)) > 6||@||
degrees while also discouraging selection of points that are similar to each other.
12:
until training converges or budget exhausted
In the training phase, we perform model training with GD (Line 10) for fixed number of iterations or until the eNTK has change too much (Line IT]. Inspired by Lee et al. (2019), we quantify this
change based on the Frobenius norm of the difference of the eNTK since the start of the phase. This is to account for the eNTK evolution during GD (Process 2 in Sec. 4.2). The point selection phase will then repeat with co-copeeeed eNTK. In practice, rather than replacing the entire training set during the new point selection phase, we will randomly retain portion of the old set and fill the remaining budget with new points selected based the recomputed eNTK. We discuss these mechanisms further in Appendix I.2
EXPERIMENTAL RESULTS AND DISCUSSION
6.1 EXPERIMENTAL SETUP
Dataset. For comparability with other works, we conducted experiments with open-sourced data Takamoto et al. 2022; Lu et al., 2021). and experimental setups that matches past work (Raissi al. (019)). The specific PDEs studied and details are in Appendix
NN Architecture and Implementation. Similar to other PINNs work, we used fully-connected NNs of varying NN depth and width with tanh activation, both with and without LAAF (Jagtap et al. 2020a). We adapted the DEEPXDE framework (Lu et al.] (021) for training PINNs and handling JAX (Bradbury et al. 2018) for more efficient computation.
input data, by implementing
Algorithms Benchmarked. We benchmarked PINNACLE with suite of non-adaptive and adap- tive CL. point selection methods. For our main results, apart from a baseline that randomly selects points, we compare PINNACLE with the HAMMERSLEY sequence sampler and RAD algorithm, the best performing non-adaptive and adaptive PDE CL point selection method based on Wu et al. 2023). We also extended RAD, named RAD-ALL where IC/BC points are also adaptively sampled in the same way as the PDE CL points. We provide further details and results in Appendix J.3
6.2 IMPACT OF CROSS INFORMATION ON POINT SELECTION
We first show how PINNACLE's automated point selection across all point types are surprisingly interpretable and similar to heuristics of past works that required manual tuning. To do so, we consider two time-dependent PDEs: (1) ID Advection, which describes the motion of particles through a fluid with a given wave speed, and (2) ID Burgers, a non-linear PDE for fluid dynamics combining effects of wave motion and diffusion. We experiment the forward problem, whose goal is to estimate the PDE solution u given the PDEs with known 33 and IC/BCs but no ExP data.
Figure plots the training points of all types selected by PINNACLE-K overlaid on the NN out- put ug. at various training stages. We can make two interesting observations. First, PINNACLE automatically adjusts the proportion of PDE, BC and IC points, starting with more IC CL. points
7

---

Published as a conference paper at ICLR 2024
(a) ID Advection problem
(b) ID Burger's problem
10000
50000
230380
Sep
Step 93330
Step pececs
KDM
8.45
PDE CL Pts
CC Pts
BC CL Pts
Figure 3: Example of points selected by PINNACLE-K during training of various forward prob- lems. The points represent the various types of CL points selected by PINNACLE-K. whereas the patterns in the background represents the obtained NN solution.
ID Advection Equation, 200k steps
96044
PENGLEE
L24
L003
R533
Stops
(b) ID Burger's 's Equation, 200k steps
The solutin
Leffor Tand
Farmmeddy
RAD
PRIADES burt PHIGEEKII
S2pp
+ Liittorm Rand
RAD
RAD-NI
PRNAACE (ours)
PARACLEK Jouns)
Figure 4: Results from the forward problem experiments. In each row, from left to right: plot of mean prediction error for each method, the true PDE solution, and the PINN output from different point selection methods.
and gradually shifting towards more PDE (and BC for Advection) CL points as training progresses. Intuitively, this helps as ICs provide more informative constraints for úg in earlier training stages, while later PDE points become more important for this information to "propagate" through the domain interior and satisfy the PDEs.
Second, the specific choice of points are surprisingly interpretable. For the Advection equation experiment (shown in Figure Ba), we can see that PINNACLE focuses on PDE and BC CL points closer to the initial boundary and "propagates" the concentration of points upwards, which helps the NN to learn the solution closer to the initial boundary before moving to later time steps. This may be seen as an automatic attempt to perform causal learning to construct the PDE solution (Wang al.) 2022b Krishnapriyan et al. (021). where earlier temporal regions are prioritized in training based on proposed heuristics. The selected points also cluster around more informative regions, such as the stripes in the Advection solution. Similar observations can be made for Burger's equation (shown in Figure (b). where points focus around regions with richer features near the IC and sharper features higher the plot. These behaviors translate to better training performance (Sec. 6.3
6.3 EXPERIMENTAL RESULTS
Forward Problems. We present the results for the 1D Advection (Figure 4ai and 1D Burger's (Figure 4b) forward problem experiments. Quantitatively, both variants of PINNACLE are able to achieve a lower predictive loss than other algorithms. The performance gap is also large enough to be seen qualitatively, PINNACLE variants learn significantly better solutions. This is especially
8

---

Published as a conference paper at ICLR 2024
evident in ID Advection (Figure 4ap, where PINNACLE learn all stripes of the original solution while the benchmarks got stuck at only the first few closest to the ICs. Additional forward problem experiments and results be found in Appendix K.2 For example, for different problem setting where we provide more training points to other algorithms to achieve the same loss, we found that PINNACLE is able reach the target loss faster. We also found that PINNACLE outperforms other point selection methods even in experiments where only one CL point type needs to be selected (e.g., hard-constrained PINNs), when no cross information among different point types is present.
Inverse problem. In the inverse problem, we estimate the unknown parameters of the PDE B, given the form of the PDE, IC/BC conditions, and some ExP points. We consider the 2D time-dependent incompressible Navier-Stokes equation, where we aim to find two unknown parameters in the PDE based on the queried EXP data. This more challenging problem consisting of coupled PDEs involving 3 outputs each with 2+1 (time) dimensions. [] that PINNACLE can recover the correct inverse parameters, while the other algorithms had difficulties doing so. Further results for the inverse problem can be found in Appendix K.3 For example, demonstrate that its quicker convergence also allow PINNACLE to achieve shorter computation time compared to other al- gorithms which are given larger training budget to achieve comparable loss. We also showed how PINNACLE outperforms benchmarks for other complex settings, such as the inverse Eikonal equation problem where the unknown parameter is 3D field rather than just scalar quantities.
Transfer Learning of PINNs with Perturbed IC. We also consider the setting where we al- ready have a PINN trained for given PDE and IC, but need to learn the solution for the same PDE with a perturbed IC. Different IC/BCs correspond different PDE solutions, and hence can be viewed as different PINN tasks. To reduce cost, we consider fine-tuning the pre-trained PINN given much tighter bud- get (5x fewer points) compared to training an- other PINN from scratch for the perturbed IC. Figure 6 shows results for the ID Advection equation. Compared to other algorithms, PIN- NACLE is able to exploit the pre-trained solu- tion structure and more efficiently reconstruct the solution for the new IC. Additional transfer learning results can be found in Appendix K.4
1.04
0.015
1.07
0.010
1.00
0.96
50000 Stops
100000
0.000
50,00
100900
Stops
Unioom
(oust
Hommersley
Figure 5: The two inverse parameters learned in the 2D Navier-Stokes equation problem. The black horziontal lines represents the true param- eter values, while the other lines represent the predicted values of the inverse parameters during training.
Inteal
Hemmendaa
PARACLESS
IMMRALE (ours)
Moan drer
30-5
656)
6306
-2533
100000 Stops
200000
Uhiform Rand
Hanmmersloy
RAD
(cui)
PANACLEK
Figure 6: Predictive error for the ID Advection transfer learning problem with perturbed IC. From left to right: plot of mean prediction for each method, the true PDE solution, and the PINN output from different point selection methods.
7 CONCLUSION
We have proposed a novel algorithm, PINNACLE, that jointly optimizes the selection all training point types for PINNs for more efficient training, while automatically adjusting the proportion of collocation point types as training progresses. While the problem formulation in this work focused on PINNs, PINNACLE variants a composite loss function and input points from different domains as well. Future work could also involve extending PINNACLE to other settings such deep operator learning.
also be applied to other deep learning problems which involve

---

Published as a conference paper at ICLR 2024
8 REPRODUCIBILITY STATEMENT
The required assumptions and proof for Theorem and Proposition 1 have been discussed in Ap- pendix and Appendix Heeiiixxeeeeppp respectively. The detailed experimental setup have been described in Ap- pendix J The code has been provided in https: /ttg//gtttuuee com/ with exception PDEBENCH benchmarks which can be found on https:://iithu/ com/ pcdbenchh/PEBBechh
ACKNOWLEDGEMENTS
This research/project is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG-PhDD2023-0110339) This research is part of the programme DesCartes and is supported by the National Research Foundation, Prime Minister's Of- fice, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. In Proc. ICMI., pp. 322-332, May 2019a.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Proc. NeurlPS, pp. 8141-8150. December 2019b.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. In Proc. ICLR, April 2020.
Umair bin Waheed, Ehsan Haghighat, Tariq Alkhalifah, Chao Song, and Qi Hao. PINNeik: Eikonal solution using physics-informed neural networks. Computers Geosciences, 155:104833, Octo- ber 2021. ISSN 0098-3004. doi: 10. cageo/........
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao Zhang. JAX: Composable transformations Python+NummPy programs, 2018.
Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George E. Karniadakis. Physics- Informed Neural Networks for Heat Transfer Problems. Journal of Heat Transfer, 146(6))06080,,, June 2021. ISSN 0022- 14811, 1528-8943.
Yuan Cao and Quanquan Gu. Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks. Proc. NeurIPS, volume 32. Curran Associates, Inc., 2019.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature Communications, 12(1):6136, December 2021. ISSN 2041-1723. doi: 10.1038/ s4 541467-021-264344
Pao-Hsiung Chiu, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, and Yew-Soon Ong. CAN-PINN: A fast physics-infommed neural network based couplelaattomaii-nuneeeeae differentiation method. Computer Methods in Applied Mechanics and Engineering, 395: 14909, May 2022. ISSN 0045-7825.
Junwoo Cho, Seungtae Nam, Hyunmo Yang, Seok-Bae Yun, Youngjoon Hong, and Eunbyung Park. Separable Physics-Informed Neural Networks. (arXiv:2306.I) 15969), July 2023.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientifie Machine Learning Through Poysics-Informn Neural Net- works: Where and What's Next. Journal of Scientific Computing, 92, July 2022.
10

---

Published as a conference paper at ICLR 2024
Zhongxiang Dai, Yao Shu, Bryan Kian Hsiang Low, and Patrick Jaillet. Sample-then-optimin batch neural Thompson sampling. In Proc. NeurIPS, 2022.
Zhongxiang Dai, Gregory Kang Ruey Lau, Arun Verma, Yao Shu, Bryan Kian Hsiang Low, and Patrick Jaillet. Quantum Bayesian Optimization. In Proc. NeurIPS, 2023a.
Zhongxiang Dai, Yao Shu, Arun Verma, Flint Xiaofeng Fan, Bryan Kian Hsiang Low, and Patrick Jaillet. Federated neural bandits. In Proc. ICLR, 2023b.
Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation eroor for physics- informed (and) operator learning. In Proc. NeurIPS, volume 35, pp. 10945-10958, 2022.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian Active Learning with Image
Data. In Proc. ICML, pp. I 1183-1 192, July 2017.
Wenhan Gao and Chunmei Wang. Active Learning Based Sampling for High- nigh-DDimensionl Non- linear Partial Differential Equations. Journal of Computational Physics, 475(C), February 2023. ISSN 0021-9991. doi: 10. jop.01./.....2222.11113
Yihang Gao, Yiqi Gu, and Michael Ng. Gradient Descent Finds the Global Optima of Two-Layer Physics-Informed Neural Networks. In Proc. ICML, pp. 10676-10707, July 2023.
Roman Garnett. Bayesian Optimization. Cambridge Univ. Press, 2022.
Alex Gittens. The spectral norm error of the naive Nystrom extension. 1 October 2011.
Alex Gittens and Michael Mahoney. Revisiting the Nystrom method for improved large-scale ma- chine learning. In Proc. ICML, volume 28, pp. 567-575, June 2013.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian Deep Ensembles via the Neural Tangent Kernel. In Proc. NeurlPS, volume 33, pp. 1010-1022. Curran Associates, Inc., 2020.
Apivich Hemachandra, Zhongxiang Dai, Jasraj Singh, See-Kiong Ng, and Bryan Kian Hsiang Low. Training-fie free neural active learning with mitialization-rooussie guarantees. In Proc. ICML, pp- 19931-12971, July 2023.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Proc. NeurIPS, volume 31, 2018.
Ameya D. Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Locally adaptive activation func- tions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 476(22299:20200334, July 2020a.
Ameya D. Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-iformmed neural networks. Journal of Computational Physics,
March 2020b. ISSN 0021-9991.
Ameya D. Jagtap, Zhiping Mao, Nikolaus Adams, and George Em Karniadakis. Physics-informed neural networks for inverse problems in supersonic flows. Journal of Computational Physics, 466: 11402, October 2022. ISSN 00219991.
Xinchao Jiang, Xin Wanga, Ziming Wena, Enying Li, and Hu Wang. An E-PINN assisted practical uncertainty quantification for inverse problems. (arXiv:2209.101) September 2022.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. Proc. NeurIPS, volume 32, 2019.
Dmitry Kopitkov and Vadim Indelman. Neural Spectrum Alignment: Empirical Study. Artificial Neural Networks and Machine Learning, 12397:168- 2020.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. In Proc. NeurlPS, vol- 2021.
ume 34, pp.
11

---

Published as a conference paper at ICLR 2024
Isaac E. Lagaris, Aristidis Likas, and Dimitrios I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):887- 1000, 1998. doi: 10.1109/22.712178
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradient Descent. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124002, December 2019. ISSN 1742-5468.
Shuning Lin and Yong Chen. A two-stage physics-informett informed neural network method based on con- served quantities and applications in localized wave solutions. Journal of Computational Physics, 457:11 1053, May 2022. ISSN 00219991. doi: 10.101 (0.1006/j..ep.22222... 1053.
Lu Lu, Xuhui Meng, Zhiping Mao, and George E. Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208-228, January 2021. ISSN 1445-1445, 1095-7200.
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs. (arXiv:2006.1114 161444440..11,,, September 2021.
Mohamad Amin Mohamadi, Wonho Bac, and Danica J. Sutherland. Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels. In Proc. NeurlPS, 2022.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning (Second Edition). Adaptive Computation and Machine Learning series. MIT Press, 2018.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Mcidani. Efficient training of physics- informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 36(8):962-977, August 2021. ISSN 1093-9687, 1467-8667.
Wei Peng, Weien Zhou, Xiaoya Zhang, Wen Yao, and Zheliang Liu. RANG: A Residual-based Adaptive Node Generation Method for Physics-Infeee Informed Neural Networks. MrXii:2205.010511 May 2022.
Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informet neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-701, February 2019. ISSN 0021-9991.
Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep active learning for image classification. In Proc. ICIP, pp. 3934 3938, Beijing, September 2017. ISBN 978-1-5090-2175.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. A Survey of Deep Active Learning. ACM Computing Surveys, 54(9): 180:1-180:40 October 2021. ISSN 0360-0300.
Simiao Ren, Yang Deng, Willie J. Padilla, Leslie Collins, and Jordan Malof. Deep Active Learning for Scientific Computing in the Wild. (arXiv:2302.00098) January 2023.
Alexander Rodríguez, Jiaming Cui, Naren Ramakrishnan, Bijaya Adhikari, and B. Aditya Prakash. Einns: Epidemiologically informedd neural networks. In Brian Williams, Yiling Chen, and Jen- nifer Neville (eds.), Proc. AAAI, pp. 14453-14460, 2023.
Franz M. Rohrhofer, Stefan Posch, Clemens Gößnitzer, and Bernhard C. Geiger. On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Informed Neural Networks. (arXiv:2203.13649) February 2023.
Bernhard Schölkopf and Alexander Johannes Smola. Learning with Kernels: Support Vector Ma- chines, Regularization, Optimization, and Beyond. Adaptive Computation and Machine Learning Series. MIT Press, Cambridge, Mass., reprint. edition, 2002. ISBN 978-0-262- 2-2--22-22997-444 978-0-
262-53657-8.
12

---

Published as a conference paper at ICLR 2024
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proc. ICLR, 2018.
Burr Settle.. Active Learning. Springer International Publishing, Cham, 2012. ISBN 978-3-031- 00432-2 978-3-031-00156--1
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1): 175, 2016.
Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Ooi, and Bryan Kian Hsiang Low. NASI: Label- and Data-agnostic Neural Architecture Search at Initialization. In Proc. ICLR, 2022a.
Yao Shu, Zhongxiang Dai, Zhaoxuan Wu, and Bryan Kian Hsiang Low. Unifying and Boosting Gradient- Training-Free Neural Architecture Search. In Proc. NeurIPS, 2022b.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz. Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: An Extensive Benchmark for Scientific Machine Learning. In Proc. NeurIPS, volume 35, pp. 1596-1611, 2022.
Kejun Tang, Xiaoliang Wan, and Chao Yang. DAS-PINNS: A deep adaptive sampling method for solving high-dimensional partial differential equations. Journal of Compulational Physics, 476: 111868, March 2023. ISSN 0021-9991.
Sattar Vakili, Michael Bromberg, Da-Shan Shiu, and Alberto Bernacchia. Uniform Generalization Bounds for Overparameterized Neural Networks. CoRR, hbs/2109.06099, 2021.
Haonan Wang, Wei Huang, Ziwei Wu, Hanghang Tong. Andrew J. Margenot, and Jingrui He. Deep Active Learning by Leveraging Training Dynamics. In Proc. NeurIPS, volume 35, pp. 25171- 25184, December 2022a.
Shusen Wang, Alex Gittens, and Michael W. Mahoney. Scalable Kernel K-Means Clustering with Nyström Approximation: Relative-Emr Bounds. Journal of Machine Learning Research, 20(1):
431-479, January 2019. ISSN 1532-4435.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of Fourier feature net- works: From regression solving multi-scale PDEs with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 384:1 13938, October 2021a. ISSN 00457825.
Sifan Wang. Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informet neural networks. (acXIv:2003.07404) March 2022b.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kemnel perspective. Journal of Computational Physics, 449:1 10768, January 2022c. ISSN 0021- 9991.
Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Transactions on Knowledge Discovery from Data, 9()), February 2015.
ISSN 1556-4681.
Zhilei Wang, Pranjal Awasthi, Christoph Dann, Ayush Sekhari, and Claudio Gentile. Neural Active Learning with Performance Guarantees. In Proc. NeurlPS, volume 34, pp. 7510-7521, 2021b.
Christopher Williams and Matthias Seeger. Using the Nyström Method to Speed Up Kernel Ma- chines. In Proc. NeurlPS, volume 13, 2000.
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non- adaptive and seiduall-assed adaptive sampling for physics-ifformed neural networks. Computer Methods in Applied Mechanics and Engineering, 403:1 15671, 2023. ISSN 0045-7825.
Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. DAVINZ: Data Valuation using Deep Neural Networks at Initialization. In Proc. ICML, pp. 24150-24176. June 2022.
Shaojie Zeng, Zong Zhang, and Qingsong Zou. Adaptive deep neural networks methods for high- dimensional partial differential equations. Journal of Computational Physics, 463:11 Au- gust 2022. ISSN 0021-9991.
13

---

Published as a conference paper at ICLR 2024
A
NOTATIONS AND ABBREVIATIONS
Table 1: List of notations used throughout the paper
Symbol
Mcaning
Example
Dirac-Delta function, I.C., وú, 1 i J and 0 otherwise
Identity matrix
N
PDE operator
IC/BC operator
X
Input space of PDE solution
0.''
Boundary of the input space of PDE solution
PDE true solution
PINN used to approximate PDE solution
C (ūg; X)
Loss of NN üg w.r.t. set X
Learning rate Input of PDE solution, i.e., elements in A' Generic training point label, i.e., E {s, p, b} Specific training point label Augmented space
8, p.
Z
Set of elements from the augmented space
Point in augmented space General prediction operator eNTK of NN iæo at timestep t ith largest eigenvalue of O
F
A
Eigenfunction of θ corresponding eigenvalue A Jacobian of (x: with respect to 0 at 9/ Residual time at point Change in residual at time at point when trained with Z
Voús, (z)
(2)
Ro. (2; Z)
Or,i (t.((Z)
Residual expansion coefficient dimension at time L RKHS of the NTK
Ho
He
Inner product of RKHS space Norm of RKHS space, i.e., Il H) Subset of Z for Nystrom approximation Subset of for point selection
Ho
Zret
Zpool
Algorithm
Table 2: List of abbreviations used throughout the paper in alphabetical order Abbreviation
Meaning
BC
boundary condition
collocation (as in collocation points)
DL
deep learning
EXP
experimental (as in experimental points)
eNTK
empirical neural tangent kernel
GD
gradient descent
IC
initial condition
NN
neural network
NTK
neural tangent kernel
PDE
partial differential equation
PINN
physics-informa informed neural network reproducing kernel Hilbert space with respect to
RKHS
w.r.t.
B RELATED WORKS
Multiple works have examined the challenges of training PINNs, such as failure of PINNs to learn certain PDE solutions, particularly those with high frequency (Krishnapriyan et al., 2021 Rohrbofer et al.) 2023 In response, there have been several proposals to adjust the training procedures of
PINNs to mitigate this effect. This includes reweighing the loss function (Wang et al. 2022c), causal
learning (Wang al. 2022b) curriculum learning approaches (Krishnapriyan al. 2021). Most
14

---

Published as a conference paper at ICLR 2024
relevant to this paper are works that looked into training points selection to improve PINNs training performance, though they have focused only on better selection of PDE CL points rather than joint optimization of the selection of ExP points and all types of PDE and IC/BC CL points. These works either adopt a non-adaptive approach (Wu et al. 2023). ) or an adaptive approach involving scoring
function based on some quantity related the magnitude of the residual each training point (Lu
et al. 2021; Nabian et al. 2021 2023
Wu et al. 2023 Peng et al. 2022: Zeng et al. 2022
Tang et al.
Outside of PINNs, the most similar problem to training point selection for PINNs would be the problem of deep active learning, which a well-studied problem (Settles 2012
Ren et al. 202T).
with proposed algorithms based on diversity measures (Sener & Savarese 2018 Ash et al. 2020
Wang & Yel 2015) and uncertainty measures (Gal et al.] 2017; Kirsch et al. 2019). Active learning
can be viewed as exploration-focused variant of Bayesian optimization (BO), which also utilizes
principled uncertainty measures provided by surrogate models to optimize black-box function
(Garnett 2022 Dai al. 2023a: Shahriari al. 2016).
number of works have also incorporated NTKs to estimate the NN training dynamics to better select points for the NN (Wang et al., 2021bb 2022a Mohamadi et al., 2022 Hemachandra et al.) 2023). However, most of these deep active learning methods are unable to directly translate to sci- entific machine learning settings (Ren et al.,
2023). Nonetheless, there have been limited works regarding active learning for PINNs, such as methods using the uncertainty computed from an en- semble of PINNS (Jiang et al. 2022).
In this work, we extensively used NTKs to analyze our neural network training dynamics. NTKs are also useful other tasks such data valuation (Wu al. 2022), neural architecture search (Shu al. 2022a b), multi-armed bandits (Dai et al. 2023b). Bayesian optimization (Dai et al. 2022). and uncertainty quantification (He et al., (2020). NTKs have also been used study NN generalization loss (Cao & Gu, 2019) and also in relating NN training with gradient descent and kernel ridge regression (Arora et al. 2019a Vakili et 202T.
C BACKGROUND ON CEYSICS-INFORMENR NEURAL NETWORK
In this section we motivate and provide some intuition for the loss function used in the training of PINNs for readers who may be unfamiliar with them.
In regular NN training, our goal is to train model such that its output is as close to the true solution as possible. This can be quantified using mean squared error (MSE) loss between the true output u(x) and NN output üs (x)) at some selected samples, which can simply be written as >x, (ūo(z))
u(x))² ignoring constant factors.
For PINNs, we also want further constrain our solution such that they follow some PDE with IC/BC conditions such as (T. This can be seen as constraint on the NN output, where we want the expected residual to be zero, i.c. considering only the PDE without the IC/BC, the constraint would be in the form N [ioo] (x) f(x) = O. However, since enforcing a hard constraint, given by the PDE, on the NN difficult to do, we instead incorporate soft constraint that introduce a penalty depending on how far the NN diverges from valid PDE solution. We would want this
penalty (or regularization term in the loss function) to of the form of E, (A[ūo](>=> ((=))) "]. where instead of performing the expectation exactly we sample some CL points to approximate the PDE loss term. This is a motivation behind using non-adaptive CL point selection method, which is seen as way to approximate the expected residual, akin to the roles of quadrature points used in numerical integration.
Alternatively, some works have considered an adaptive sampling strategy where the CL points are selected where there is a higher residual. In this case, the expected residual plays a different role, which is encourage the NN to learn regions of the solution with higher loss.
D BACKGROUND ON NEURAL TANGENT KERNELS
PINNACLE relies on analyzing NN training dynamics and convergence rates. To do so we require the use of Neural Tangent Kernels (NTKs), which is tool which can be used to analyze general
15

---

Published as a conference paper at ICLR 2024
NNs well as PINNs. The empirical NTK (eNTK) of NN ăe step of the training process Jacot et al. 2018),
(x, Vgūe, Voúo,
(16)
can be used to characterize its training behavior under gradient descent (GD). For NN trained using GD with learning rate 7/ on training set X, the NN parameters 0g evolves as 0c+1 VV X) while the model output evolves as ui 1 (Ir) iig,, (x) ,0, X dg ( X).
Given MSE loss [(ūg; X) = Emex (ūo(a) u(x))² 2 / the change in model output during training in this case can be written as
Adg, (x)
Úg (x) úo (x) - 1 (ao, (X) u(X))
(17)
Past works (Arora et al. learning rate, the NN follows a linearlized regime and the eNTK converges to fixed kernel that is invariant over training step t. Even though these assumptions may not fully hold in practice many works (e.g.. Arooa et al. a); Lee et al.] (2019); [Shu et al. (2022b)) have shown that the NTK still provides useful insights to the training dynamies for finite-widh wiith NNs.
201 had shown that in the inddnite-width limit and GD with infinitesimal
E BACKGROUND ON REPRODUCING KERNEL HILBERT SPACE
In this section, provide a brief overview required on the reproducing kernel Hilbert space (RKHS) for our paper. This section adapted from Schölkopf & Smola (2002).
Consider symmetric positive semi-definite kernel K. : XXXx X + R. We can define an integral operator Tx: as
Txlf()) [ ((x))(zx)))(xx))
(18)
We can compute the eigenfunctions and corresponding eigenvectors of Tx. which are pairs func- tions vi; and non-negative values A; such that Tx[+:](+) A-vh(-). Typically, we also refer the eigenvectors and eigenvalues of Ti as the eigenvectors and eigenvalues of IC. By Mercer's theorem, we can show that the kernel K. be written terms of its eigenfunctions and eigenvalues as
((xxx') (z)a())
(19)
Based on this, we define the feature map as
e,(+) = -(x,, Σ (x)x())
and consequently the inner product such that (vh, dyyne = Se.g As- This is so that able to write
(0(z),
(20)
(2( Σ Anti()) = K(xxx').
(21)
From this, the RKHS Hx a kernel K. would be given by a set of function with finite RKHS norm, i...
Hx. { (
(J, Ws
S soo
(f.1
(22)
F EXTENDED DISCUSSION FOR SEC. 4.2
The contents in this section has been adapted from NTK related works (Lee et al. 2019) to match the context of NTK for PINNs.
16

---

Published as a conference paper at ICLR 2024
Calculations of (8). From the gradient descent update formula with loss function given by (
can see that
Oe+1 O2 = -1 VoL(ug 2)
(23)
V Fleg (2)
T V F(ao) ((ie ?)
by chain rule
(24)
- [e,, Rg, (Z)
(25)
which, as 1) + O, can be rewritten with continuous time GD (where the difference between each time step is approximated as derivative) as (Lee et al. 2019)
0,, = F
Ra, (Z)
(26)
and so under continuous time GD, we can write the function output as
Aoo (z; Z) = 0 Re, (z; Z)
as 17 - 0
(27)
0 (2)
(28)
Fio (z) 0.0:
by chain rule
(29)
F Fūo, ((=) V Flo,oo,122 (Z) Ro, (Z)
(30)
= =n)0r(2, 2) Ro. (Z).
(31)
Calculations of (9) In this case, we consider the PINN under the linearized regime, where we approximate F(uoo] using the Taylor approximation around 0o.
Fllm [20] (2) = F [2)) + Vo F( (0 6o).
(32)
and subsequently Rijn = Flim [o) (2) (2). Under this regime, the eNTK remains constant all i, i.e., O,(2, z') = 0'( Vo (o [2)) VoL"(oo,(==') (Lee et al. 2019). which also
means that its eigenvectors and corresponding values will not evolve over time. It has been shown
empirically both for normal NNs (Lee al., (2019) and PINNs (Wang et al. 2022c that this regime holds true for sufficiently wide neural networks.
For simplicity we let v/s and A; be the eigenfunctions and corresponding eigenvalues O₂ (where we drop the subscripts t for simplicity). The change in residue of the training set can be estimated using continuous time GD Altli 2): ~ O, (2; 2), and based on (31 can be rewritten as
0, 0, = -1000() ( (Z)
(33)
which, when solved, gives
Rlin (Z) c ( Rlin (Z).
(34)
To extrapolate this result to other values of 2 Z, we use the definition of (Z) to that
Flin (ie) [Z)](Z)= e (7)) lin (2) Flu((2))
(35)
and therefore Flin [ūo, (Z) = Flu + t (10((2 (F (Z)) - Flu](Z))
(36)
(F (11-e-1t0((2)) [uo.] 1 (Z) Flui(Z))
(1
(2))
(37)
(uo (2)) 0o(2)0o(2) - (100((2) R!ii (2)
(38)
Frin (2) VoFluo,112 VoFlue, O (2) (110 - 1)
lin (Z)
8o
0.-60
(39)
which, through comparing the above with the form of the linearized PINN, gives what 0, 0 is
during training. Therefore, plugging this into 32
[ (i2) Flin (2) + V
V o 04,
Fl
(Z) ((00(())) Rli am
(40)
(o. [o) (W011) - -1) -1) (Z)
(41)
Rlling (2; [2 |lll (2; 2) (e-119 1) 1eliin (2).
(42)
17

---

Published as a conference paper at ICLR 2024
We can then decompose (42) into
Rby (2; 2) Ro (2; 0 Z)
ws (2)
T Rag (Z).
(43)
A
Eigenspectrum of the eNTK. In this section, we elaborate further about the eigenspectrum of the eNTK. In Figure 7 we plot out the eigenfunctions 14x, of the eNTK for the ID Advection forward problem at GD training steps t = 10k and t 100k. We can see that in the case the PINN output úg is already able reconstruct the PDE solution well), the eigenfunctions able to reconstruct the patterns that correspond to the true solution well. Note that due to the tight coupling between the true NN output and the PDE residual, the components of the eigenfunction for (r,8) and (I, p) are correlated and may share similar patterns. The dominant eigenfunctions will then tend either have e (2, s) which aligns well with the true solution, or have (2, p) which is flat (corresponding to the true PDE solution whose residual should be flat). In the less dominant eigenfunctions, the patterns will align worse with the true solution. Meanwhile, in the case, we see that the eigenfunctions will share some pattern with the true solution, but will not be as refined as for 100k. Furthermore, the less dominant eigenfunctions will be even noisier.
= 100k (when
10k
10k Steps
Step 10000
(b) 100k Steps
Step 100000
MR ELEret
2.90.01
Juye 6.42
Figure 7: Eigenspectrum of O, at various timesteps.
In Figure 8 we show how the eigenfunctions can describe the model training at given GD training step. We see that by using just the top dominant eigenfunctions we are able reconstruct the NN output uz.
(a) After 10k steps of GD
(b) After 100k steps of GD
Thoo soluio
Top uggh
Top th
500 do
Toe solutiii
Top 10 dg... The og
-333
1210
2004
Figure 8: Reconstruction of the NN prediction using the eigenbasis of O
18

---

Published as a conference paper at ICLR 2024
G PROOF ON GENERALIZATION BOUNDS
In this section, we aim to prove Theorem | regular NNs to give a generalization bound for PINNs. In order to do so, we will first state some assumptions about the PDE being solved and the PINN used for the training process.
The proof is an extension Sf et al. (2022b) beyond
Assumption 1 (Stability of PDE). Suppose we have differential operator N P - y where P is a closed subset of some Sobolev space and y is a closed subset of some L space. For any u, v E P, let there exists some Cpde 2 1 such that
[ - MTully S Optel luu ul|p.
(44)
For simplicity, we can view the Sobolev space as I space with a defined derivative up a certain order. Note that while it is possible for Cpte < 1, we assume otherwise for convenience.
We now provide an assumption about the NN we are training with.
Assumption 2 (On the Neural Network Training). Assume some NN ûg X - R with L hid- den layers each of width n.. Suppose there exists constants p such that [[ S and || aN NNooo (x)] S p. Also let Imin and Umax be the minimum and maximum of the eigenval- ues of the eNTK matrix of the possible training set at initialization, ie., for any training set S, bmax 2 (1(((()) 2 ... >1ss)(Oo 2 bmin-
We will also make an assumption about the boundedness of the PDE solution u and the NN output úg. This is done for convenience of the proof - it is possible to add appropriate constants to the subsequent proofs to get a more general result.
Assumption 3 (Boundedness of the NN Residual). Suppose the PDE solution and the NN output was such that R (2, s) I 1/Cpde- This would also imply that |Ro (z. (.) S 1.
We will be working with the hypothesis spaces
u a f (1)o.(2)):tt
: +t
(45)
and
lin {z F( + FeFlúa, (() (0, - -0) :t> 0}
(46)
which is the linearization of the operated neural network F(ūo] around the initialization given by (z)) se fell (2) = F F(ae,,[=)-VV V F
[=) (0--0)). Note that since f performs a Taylor expansion with respect to 0, this means the operator can only be applied on functions approximated by NN that is parametrized by parameters A. This is unlike F which is an operator that can be applied on any function. Also note that the linear expansion performed on Flúo] (rather than on just the operand úg), and hence does not require the assumption that F is linear operator. We can also define each components of plin [Uo] separately as
1g (xx)
p pli (2)] s) uo. (x)
V Vouo (x) (0. 0o),
(47)
and
(x) A [ (2., p) (x) + VoN ((ooool (x) (0:- -0))
(48)
First we will state the result for training the NN where the loss function is based on the linearized operator, or
( Z)
Σ (Ftt - Flu((x))²
(49)
=EZ
We will show that under the linearized regime, the weights of the NN will not change too much compared to its initialization.
Lemma 1. Consider a linearized neural network a/jm following Assumption is trained on training
set Z C Z with loss in the form (4) with GD with learning rate < 1/ Amax (0o (2)). Then, for
any >
100 .oll < =10000 - 0oll v (Z),
(50)
19

---

Published as a conference paper at ICLR 2024
Proof. Let Rlj Fllm [uel (2) (2) be the residual of the linearized operated network. Note that Rilinn (2) Reg (z). Based on the update rule of GD, we see that
8 = 17 VaF(ūo,1(Z) R (Z)
(51)
and so
Flin ( = Flug,11 (2) + VoF (2) - 80)
(52)
= (( + [ot (2) 17 V VeFlāg,1(2) (0,(()) - 0o)
(53)
1 (ūe, 77. (2)
(54)
(1 -1000) Flin (0 (2) + ,10
(55)
(1 n)))) 1+1 Rlge (Z) + Flu(Z))
(56)
where is the identity matrix and the shorthand O = Oo(Z). We note that (56) arrives from
applying (55) recursively + times and using the fact that we set 17 < 1/ Amux (Z) simplify
a corresponding geometric series. This gives
0, 00
(0++1 0%)
(57)
k=0
= 7) V (1000ol
Rhm (7)
(58)
R=
t
=1- VoF U 1 (Z)
Σ
(1 - m70))
1+1
Rig (Z).
(59)
k=0
The remainder of the proof follows directly with Lemma A.5 from Shu et al. 2022b).
Corollary 1. Given Lemmal 7
Amax (Oo(S))
110 - -oll s (S)11?
(60)
(SS) Amin (s))))
Proof. Notice that we can rewrite a(5) as
a(S)= = Σ
Re, RR, Σ
(s) Ro. (5))
(61)
Ho.
Σ
(S)
(62)
(S)
Σ Xe Ne. (S)
(63)
Re. (S)))))))
(64)
Ro, (S) (S)
(65)
where we use θ = O, due to the linearized NN assumption. Let A = Oo(S), y = (S) and
k = |R, (S) Note that A is a PSD matrix, and therefore we can define some A 1/2 such that
A = (A1/2)(( (A1122) and also some A 1/22 such that A = (A-1/2) (A-1/2).
y Ay) A A y) //2 v11 11 ||/-
1/2 y/1
(66)
< x A 112 x² ( 1
(67)
no Amax (A1/2)2 Ammax (A-2/2
(68)
n Amna (A) Amax (A 1A-1)
(69)
n Amax (A)
(70)
Ami (A)
20

---

Published as a conference paper at ICLR 2024
which gives
Amax (b)
Ra. (5))) -R, (R) - |R, (S)]1,
(71)
a{S)) Amin (Oo(S))
We now will link the linearized operated NN the non oon-linearized version. We first demonstrate that the difference of the two versions of the NN can be bounded.
Lemma 2. Suppose we have neural network úg such that Assumption 2 holds. Then, there exists some constants c, cd,, NL, Nlo > such that if n > NL. then when applying GD with learning
rate < 1jo.
sup 120
lin úo.1l S 10g
(72)
VnL
and
C. Cpde + d' |Ro, (5)1l
Amax (0o(S))
sup
[Lt.il -V &
(73)
t20
VTL
a(S) Amin (n(S))
with probability at least over the random network initialization.
Proof. The first part follows from Theorem H.1 Lee et al. (2019). The second part follows the fact that
|| [xr))xx) [(]o)
(74)
I (x) + VoNViu, (x) (0, - 00) - 1x)o,(())
(75)
S ILV - . V(@o.I<)))
(0. - -b)ill
(76)
Cpae (0, (x) 4o. (x)] VVV [(:: - -bo)ll
(77)
Opde (1100, (x) (z)] ||]] 02, (z)]l) [VVoN (0) - Bu)
(78)
Opac (II 0 (ig (x)T(B) 0u)l || (z) 0e. (=) + V VoNTuo, (xx)
T() (0. 0o)
(79)
Opde /| Van (0. lo)l + Va.Niin, (0. - 0o)1 + Gpdee 1b (2:) úo. (x)
(80)
(Cpde VoN + || / [1 Ooll + Opde
(81)
V
Amax (S) + Cpde
(1 + Ro
(82)
(S)) Amin (Qn(S)
VTL
which proves the second part of the claim. Note that the bound from
73 could be used to bound
72 as well.
We now use results above to compute the Rademacher complexity of U. We first compute the Rademacher complexity of ii in Lemma 3 then relate it to the Rademacher complexity of U in Lemma As reference, the empirical Rademacher of a hypothesis class & over set S is given by (Mohri et al. 2018)
e (()))
R9( A Bre(L1his)
sup gEg
(83)
[S]
Lemma 3. Let Rg( the Rademacher complexity of hypothesis class G over some dataset with size Ns. Then, there exists some O such that with probability least 1 - 8 over the random initialization,
R)( <
P
(S)I?
Amn (Qo(S)
(84)
Na/2
a(S) Amiin (Q)
21

---

Published as a conference paper at ICLR 2024
Proof. We apply the same technique from Shu al. 2022b). We that
Ns.
[8,
(24
Ns
V
(2:) (0:
80)
Rs(((m)) = B
+ sup
B
(+1
Ns
Ns.
120
(85)
0
0o
V F 10 (S)
$ sup
(86)
120
Ns
1 $ Ns
|R, (S)]I2
Amax (O)(S)
P VNs
(87)
((S). Amin (Q))
In the first equality, we use the definition of R. The first part of the first equality is independent of I, and will cancel 0 due the summation over e. The second inequality then follows from Cauchy-Schwartz inequality. The third inequality uses Lemma ! and the assumption regarding |[V&F ((),),))
Lemma 4. The Rademacher complexity of is given by
Gpde VUL
(Ng/2 N/2/2 + +c) ||oo, (S)II
Ammux (Oo(S))
Rs(U))
+
V
(88)
a(S) Amin (Q)(S)
for some e and d.
Proof. From Lemma 2 we know that there will exist some constant and d' such that for any & E [+1}.
Gpde vn
Amax (On(S)
(F(ūo,](2)) S (FFllin (
+ + Ro, (S)]12.. V ((()) Amin (Q (5))
(89)
which gives
Opte
Amax (0o(S))
Rs((1) s R
d' Ro (S)1
(90)
VnL
a(S) Amin (Qo(S))
Applying Lemma to the above completes the lemma.
We now use the Rademacher complexity to relate back to the overall residual. For simplicity, we let r(z)= |Rg (2)l and let FD = [x>[(r(>)] be the expected absolute residual given test samples drawn from a distribution D.
Lemma 5. Suppose S ~ Ds is an i.i.d. sample from Z according to some distribution Ds of size Ns. Then, there exists some c. ,N such that for nu 2 NL, with probability at least 5,
2cCp Vnz.
(- P
Amax (Q)(S)
(2/8)
TD3 5 Ng
Ro (S)
+c
(S)II,
aa(S) Amin (On(S) +3/
N N3/2
2Ng
(91)
Proof. Note that based on Assumption we have Re__ (=) < 1. The result then follows from directly applying Lemma to Thm. 3.3 from Mohri al. 2018).
We now proceed to prove Theorem We will first state the theorem more formally, then provide a proof for it.
Theorem 2 (Generalization Bound for PINNs). Consider PDE Nu, ()()) f(x) follows Assumption I Let = [0, 1]d x {s.p). Let úg be a NN which follows Assumption 2 Let the NN residual follows 3ssumption Let Dg be a probability distribution over 2 with p.d.f. p(2) such that
22

---

Published as a conference paper at ICLR 2024
u(12 E 2 p(2) <1/2)) I s. where ((W) is the Lebesgue measure of set W Let ûg be trained on dataset S C 2, where S ~ Ds uniformly at random with |S|= Ns, by GD with learning rate
n S 1 (S)( Then, there exists some e such that with probability 28 over the random
model initialization and set S.
Ex (x) -u(x)]] S c Ro (S) + Va(S)
+ C3
(92)
where
2
C1 Ns'
(93)
VPV |max
bmax bmin
C2
+ + Cpde)
(94)
bmin \
4c Opte
log(2/5)
C3
+ +GV
2ss
+ 2s.
(95)
TL
2NVs
Proof. Let Dz refer to the uniform distribution over Z. The p.d.f. of the uniform distribution & is given by g(=) = 1/2. Notice that we can then write
1 B [x()]
1 [ 2 [xxx/((()))
GDz 2
(96)
Let - = 2 2 1/2} be set of input with a "high" probability of being chosen by the point selection algorithm, and 2 2 2y. Then, abusing the integral notation allow summation over discrete spaces, we see that
FD EUNDE [r())]
(97)
r(z)dz+ + { r(z)dz
(98)
21
/ r(z)do(=)
/ r(z)dz
(99)
2,
2,
[ dz
< r(z)dp()) +
(100)
2
ÎDS + +((2))
(101)
S FDg +
(102)
and so
Exax [r(x)) < 2(FDs + +))
(103)
Applying results from Lemma to bound iDy completes the proof.
H FORMAL STATEMENT AND PROOF OF PROPOSITION
In this section, we will drop the subscript / from the eNTK O for convenience.
We use slight abuse of notation to work with the augmented space Z. Given X' is defined a Cartesian product of intervals, its I Lehesgue measure is naturally defined. Then, formally, for W 2, we define ((W) fu(f X: s) W)) + fu((= X' (x,p) W)).
23

---

Published as a conference paper at ICLR 2024
We first make the following observation about the quantity á(Z) in (14). The quantity can be rewritten as
à i0 O(Zret: Ro, (2))
(104)
(Z) ((2 (2)) Znei Σ
0 ( Z) (Z)
(105)
i=1
Ro, (Z) (Z Z (Zer))
- (Zrd 2) Ro, (Z).
(106)
Notice that 3 can be seen as a Nystrom approximation of ( where the approximation is done using the inputs Zee-- We will first analyze ④ by comparing its difference from the full-rank matrix (Z)). This result is direct consequence of the approximation error bounds from previous works on the Nystrom approximation method.
Proposition Let Npost be the size of set Zpool- Let E (0, 1) be a number. Suppose we write O (Zpma)) = UAUT using the eigenvalue decomposition such that columns of U are eigenfunctions of o (ppci)) and diagonals of A are their corresponding eigenvalues. Let Ug be the first k columns Uf corresponding to the top eigenvalues of e (op))) and let - (Npoot k)) max, II(U)) be the coherence of Uk- Let & S Npoot be some integer. Then, if a set Zeef of size p where
2 log(k/8) p2
(107)
(1 E)-
is sampled uniformly randomly from mooo then with probability greater than 1 8, for any Z C 7
2)112 S * (1+
Npaal
Z,e ((2
$ c
(108)
EP
where >x (O(2oo)) is the kth largest eigenvalue of O ((pooo)).
Proof. Let D(Z) 0(Z) O(Z, Ze ( (er Z). Consider the case where 7
Then, based directly from Thm. 2 in Gittens 2011 we can show that [Z)]] Amin (0 A (Zooo))
(1 + Npoot/sp)
To apply this to every Z, note that since Z C Zpoot, it means D(Z) will be submatrix of D (1ppa)) and hence it is the case that l12 < D D(Zpaa))ll, which completes the proof.
A few remarks are in order.
Remark 1. The bound in Proposition depends the kth smallest eigenvalue of θ (Zoo)) From both empirical (Kopitkov & Indelman 2020: Wang et al. 2022c) and theoretical (Vakili et al. 2021 studies of eNTKs, we find that the eigenvalue of eNTKs tend to decay quickly, meaning as long as our pool Zpoot is large enough and we let & be large enough value, we would be able to achieve an approximation error which is arbitrarily small.
Remark 2. In Proposition 2 (and throughout the paper) we assume that the set Z is constructed by uniform random sample. As a result, the number of points e that the bound requires is higher than what would be needed under other sampling schemes (Gittens & Mahoney 2013 3 Wang et al., 2019, which we will not cover here. The version of Proposition only serves to show that as long as we use enough points in Zuer, the Nystrom approximation can achieve an arbitrarily small error in the eNTK approximation.
We now attempt to link the difference in Frobenius norm proven in Proposition 2 to the difference between a(Z) /Z) and á(Z).
Proposition 3. Let the conditions in Proposition For given Z C Zpoot, a((Z) a(Z)) s él Ro, (Z) 3
24

---

Published as a conference paper at ICLR 2024
Proof. It applies directly from Proposition that
[a(Z) - - (2)) = RR, (2) (0
0(, (())) (z (10 Ro, Ro, (2)
(109)
1 Re, (2)1 - - (()))))))) 1012, 2)
(110)
< a| Ro, (
(111)
where the second inequality follows from the definition of Frobenius norm.
I EXTENDED DISCUSSION ON PINNACLE ALGORITHM
1.1 DISCUSSION OF THE POINT SELECTION METHODS
PINNACLE-S. Theoretically, PINNACLE-S is better motivated than PINNACLE-K. This is be- cause PINNACLE-S involves sampling from an implicit distribution (whose p.d.f. proportional to â), and therefore is more directly compatible with assumptions required in Theorem I where $ assumed be i.i.d. samples from some unknown distribution.
In theory, PINNACLE-S SS is also sensitive to the ratio of the pool of points. For example, regions with many duplicate points will be more likely to be selected during sampling due to more of such points in the pool. This, however, is an inherent characteristic of many algorithms which selects batch of points using sampling or greedy selection methods. Despite this, in our experiments we select pool size that is around 4x as large as the selected points budget in each round, which encourages enough diversity for the algorithm, while still allowing the algorithm to focus in on particular regions that may help with training. Future work could be done to make PINNACLE-S more robust to pool size, through using sampling techniques such as Gibbs sampling which would make the method independent of pool size.
PINNACLE-K. For the K-MEANS++ sampling mechanism, we use Euclidean distance for the embedded vectors. This is similar to the method used in BADGE Ash al. (2020). Each dimen- sion of the embedded vector will represent a certain eigenfunction direction (where a larger value represents a stronger alignment in this direction). Therefore, points that are selected in this method will tend to be pointing in different directions (i.e., act on different eigenfunction direction from one another), and also encourage selecting point that have high magnitude (i.e., high change in residual, far away from other points that has less effect on residual change).
We demonstrate this point further in Figure 9 In Figure 9a where we plot the vector embedding representations, we see that (i) the points with smaller convergence degree (i.e., lower norm in em- bedded space) tends to be "clustered" together close to the origin, meaning fewer of the points will end up being selected; (ii) the dimensions in the embedding space that correspond to the dominant eNTK eigenfunctions (i.e., higher eigenvalues) are more "stretched out", leading to coefficients that are more spread out, making points whose embedding align with these eigenfunctions to be fur- ther away from the other points. These two effects creates the "outlier" points that are more widely spread out due to having higher convergence degrees preferentially selected. Figure poaure/blelpliittly sorts the points based on their convergence degrees, and shows that PINNACLE-K mainly selects points that have much higher convergence degrees.
1.2 TRAINING POINT RETENTION MECHANISM
We briefly describe how training points chosen in previous rounds can be handled during training. While it is possible to retain all of the training points ever selected, this can lead high computa- tional cost as the training set grows. Furthermore, past results (Wu et al. (023) have also shown that retaining all of the training point could lead to worsen performances.
Alternatively, we can have a case where none of the training points are retained throughout training. This is done in the case of RAD algorithm (Wu et al.) 2023). However, in practice, we find that this can lead to catastrophic forgetting, where the PINN forgets regions where training used to be at. As a result, in PINNACLE, we decide to retain a number of training points from the previous round when choosing the next training set. In Figures 10 and II PINNACLE consistently drops when this mechanism is not used, demonstrating the usefulness of retaining past training points.
that the performance of
25

---

Published as a conference paper at ICLR 2024
(a) Embedding of (Ai (z)) for different values of
(b) Value of á(z)
10:
Component 2
Compenent 100
Value att)
-20
0.75
102
-40
0.50
10
0.25
-60
0.00
10-3
-80
500
1500
Component
Component
Rank of al2)
Figure 9: Illustrations of the K-Means++ sampling mechanism. each plots, the blue points are the data pool, while the orange point represents the selected points by the K-Means+ mechanism. Figure (9a| plots the embedding in two of the eigenfunction components, while Figure [96] plots the range of the individual convergence degree of each points. The embedding is obtained the 1D Burger's equation problem after 100k trained steps, and the point selection process aims to select 30 training points (note that this is fewer than the actual training set selected we select fewer points to achieve a clearer illustration of the method).
1.3 TRAINING TERMINATION VIA ENTK DISCREPANCY
In PINNACLE, we also check the eNTK and perform training point selection again when the eNTK changes much. This corresponds when the chosen of training points are longer optimal given the PINN training dynamics. In Figures I0 and IT we show that auto-triggering of AL phase can help improve model training in some cases (in particular, for the Advection equation with PINNACLE-K as seen in Figure I1a). However, in practice, we also find that this is often less useful in practice, since as training progresses, the eNTK does not change by much and so the AL. process will unlikely be triggered before the maximum training duration T anyway.
1.4 PSEUDO-OUTPUTS USED FOR EXP POINT SELECTION
The residual term Rg,(2) can be computed exactly in the case for CL points since the PDE and IC/BC conditions
completely known (except for the inverse problem where the PDE constants will need to be the estimated values at that point in training), and is independent of the true solution. However, in the case of EXP points, Re, (x) will depend on the true output label, which unknown during training. To solve this problem, we approximate the true solution u by training the NN at the time of the point selection úg for T' steps beyond that timestep, and use the trained network ú, as the pseudo-output. This works well in practice as the magnitude of the residual term roughly reflects how much error the model may have at that particular point based on its training dynamics. In our experiments, we set T 100.
J EXPERIMENTAL SETUP
J.1 DESCRIPTION OF PDES USED IN THE EXPERIMENTS
In this section, we further describe the PDEs used our benchmarks.
1D Advection equation. The Advection PDE describes the movement of substance through some fluid. This is simple linear PDE which is often used to verify performance of PINNs in simple and intuitive manner due the easily interpretable solution. The ID Advection PDE is given by
Ou at
Ou
0, E E L), / E (0,7)
(112)
dr
with initial condition u(x,0) 0) = un uz) and periodic BC u(0, t) = ul (L, is used. The
closed form solution is given by u(x, t) uo(ar 3t). To compare with the PDE from (1)
26

---

Published as a conference paper at ICLR 2024
(a) ID Advection
(b) ID Burger's
10-1
Mean error
x
Mean errar
10
10
6x
100000 Steps
200000
100000
200000
Steps
Uniform Rand
No Memory
No Auto Trigger
PINNACLE-S
Figure 10: Performance of PINNACLE-S on the various forward problems when different compo- nents the algorithm are removed.
(a) ID Advection
(b) ID Burger's
Mean error
Mean error
10-2
10
10-2
100000 Steps
200000
100000
200000
Steps
Uniform
No Memory
No Trigger
PINNACLE-K
Figure 11: Performance of PINNACLE-K the various forward problems when different compo- nents of the algorithm are removed.
Dy and fxr t) = 0. Meanwhile, for the boundary conditions, the IC can be written B [M] u and gl (I, t) uo(z), while the BC be written as B2 B2[u1(x, = u00,t) - u L, t) and (zx f) 0.
we have W[u, 3] 8y
3 8u
In our forward problem experiments, we use 8 = with a irregularly shaped IC from PDEBENCH (Takamoto et al. 2022). In the timing experiment (in Figure 21a), we use / 5 with sinusoidal initial condition, past works have highlighted that the ID Advection problem with high 3 values is a challenging problem setting (Krishnapriyan et al. 2021), which is possible for the timing experiments given that we provide more CL points to the benchmark algorithms.
ID Burger's equation. Burger's equation an equation that captures the process of con- vection and diffusion of fluids. The solution sometimes can exhibit regions of discontinuity and can be difficult to model. The 1D Burger's equation is given by
Ou 1 2u
8²u
0t 2 dr = Oz2' IE (0, L), t (0,2)
(113)
with initial condition u(x 0) = (z) and periodic BC u(0 = uu L, ) is applied. In our experiments, we used the dataset from Takamoto et al. (2022) with 0.02.
1D Diffusion Equation. We aim to leam neural network which solves the equation
Ou Pu
-2 sin(A (2 4x²)
(114)
Ōt dr
with initial conditions u(x, 0) = sin (4Tx) and Dirichlet boundary conditions u( u(-1,t) = v11 t) 0. The reference solution in this case is u(x; e sin(4
27

---

Published as a conference paper at ICLR 2024
In the experiments, solve this problem by applying hard constraint (Lagaris et al. 1998). To do so, we train a NN ug x t), however then transform the final solution accord- ing to (1( t) t(1 x² (z)
sim(4xr).
ID Korteweg-de Vries (KdV) Equation. We aim to learn the solution for the equation
du
du Ou Ox.
+ X u
A2
(115)
a1
Ōr
with initial condition u(ar, 0) (OS( and periodic boundary condition u(-1,t) u(1,t) t). We solve the forward problem instance of the problem, where we set Y = 1 and A2 0.0025.
2D Shallow Water Equation. This is approximation of the Navier-Stokes equation for modelling free-surface flow problems. The problem consists of a system of three PDEs modelling four outputs: the water depth h, the wave velocities and v, and the bathymetry b. In this problem, we are only interested in the accuracy of h(,, y, t). We use the dataset as proposed by Takamoto et al. (2022).
2D Navier-Stokes Equation. This is PDE with important industry applications which is used to describe motion of a viscous fluid. In this paper, we consider the incompress- ible variant of the Navier-Stokes equation based on the inverse problem from Raissi et al. (2019), where we consider the fluid flow through a cylinder. The output exhibits a vor- tex pattern which is periodic in the steady state. The problem consists of the velocities u(æz t) and v(x, y, t) in the two spatial directions, and the pressure p(x, y, t). The three functions are related through a system of PDEs given by
au
du
Ou
Op
82
8u
+
+
0
+ A2
(116)
0t
Ōr
0y
dr
dr
oy²
Ov
u Ou
+ U 0u
Op
jpu
82v
+
A1
+ X2
(117)
Öt
Ox
0y
dy
Ox²
0y²
In the inverse problem experiments, we assume we are able to observe u and v, and our goal is to compute the constants xı and A2, as well as the pressure p(x, V. u). There are no IC/BC conditions in this case.
3D Eikonal Equation. The Eikonal equation is defined by
VT(()] = (xx)
(118)
with (xx)) 0 for some To- For the inverse problem case, we are allowed to observe ((x) different values of x, and the goal is to reconstruct the function u(x).
The budgets allocated for each types of problems are as listed below.
Equation (Problem Type)
Training steps
CL. points budget
EXP points query rate
ID Advection (Fwd)
200k
1000
ID Burger's (Fwd)
200k
300
ID Diffusion (Fwd)
30k
100
ID KdV (Fwd)
100k
300
ID Shallow Water (Fwd)
100k
1000
2D Fluid Dynamies (Inv)
100k
1000
30 per 1k steps
3D Eikonal (Inv)
100k
500
5 per 1k steps
ID Advection (FT)
200k
200
ID Burger's 's (FT)
200k
200
J.2 DETAILED EXPERIMENTAL SETUP
All code were implemented in JAX (Bradbury al. 2018). This is done due to its efficiency in performing dito--ffferentation.. To make DEEPXDE and PDEBENCH compatible with JAX, we made modifications to the experimental code. While DEEPXDE does have some support for JAX, it
28

---

Published as a conference paper at ICLR 2024
remains incomplete for our experiments and we needed to add JAX code components for NN training and training point selection.
In cach experiment, dependent on the specific setting. Each experiment uses tanh activation, with some also using LAAF (agtap et al. 2020b). The models are trained with the Adam optimizer, with learning rate of 10-4 for Advection and Burger's equation problem settings, and 10-3 for the others. We list the NN architectures used in each of the problem setting below.
multi-layer perceptron with the number of hidden layers and width
Problem
Hidden layers
Hidden width
LAAF?
ID Advection (PDEBENCH, 1)
128
ID Advection (3 5)
6
64
Yes
ID Burger's
128
1D Diffusion (with hard constraints)
2
32
Korteweg-de Vries
32
2D Shallow Water
4
32
2D Fluid Dynamics
64
Yes
3D Eikonal
8
32
Yes
The forward and inverse problem experiments in the main paper were repeated 10 rounds each, while the rest of the experiments repeated 5 rounds each. In each graph shown in this paper, report the median across these trials, and the error bars report the 20th and 80th percentile values.
For experiments where computational time is not measured, we performed model training on compute cluster with varying GPU configurations. In the experiment where we measure the algo- rithms' runtime in forward problem (i.e.. experiments trained each NN using one NVIDIA GeForce RTX 3080 GPU, and with Intel Xeon Gold 6326 CPU @ 2.90GHz, while for the inverse problem (i.e., experiments corresponding Figures 21b and 21ch. we trained each NN using one NVIDIA RTX A5000 GPU and with AMD EPYC 7543 32-Core Processor CPU.
corresponding to Figure 2la). have
J.3 DESCRIPTION BENCHMARK ALGORITHMS
Below we list some of the algorithms that we ran experiments on. Note that due to space constraints, some these experiments are not presented in the main paper, but only in the appendix.
We first list the ron-adaptive point selection methods. All of these methods were designed to only deal with the selection of PDE CL points hence IC/BC CL points and ExP points are selected uniformly random from their respective domains. The empirical performance of these methods have also been studied in Wu et al. (2023)
UNIFORM RANDOM. Points are selected uniformly random.
HAMMERSLEY. Points are generated based the Hammersley sequence.
SOBOL. Points generated based on the Sobol sequence.
The latter two non-adaptive point selection methods are said to be low-discrepancy sequences, which are often preferred over purely uniform random points since they will be able better span the domain of interest. In the main paper only report the results for HAMMERSLEY since it is the best pseudo-random point generation sequence as experimented by Wu et al. (2023).
We now list the adaptive point selection methods that have used.
RAD (Wu et al., 2023). The PDE CL points are sampled such that each point has prob- ability of being sampled proportional
the PDE residual squared. The remaining CL points are sampled uniformly at random. This is the variant of the adaptive point selection algorithm reported by (Wu et al. 2023) which performs the best.
RAD-ALL. This is the method that extended from RAD, where the IC/BC CL points are also selected such that they are also sampled at probability proportional to the square of the residual. ExP points selected using the same technique described in Appendix I.4
29

---

Published as a conference paper at ICLR 2024
which is comparable to expected model output change (EMOC) criterion used in active learning (Ranganathan et al. 2017).
PINNACLE-S and PINNACLE-K. This is our proposed algorithm as described in Sec. 5.2
In cases where there are multiple types of CL points, for all of the methods that do not use dynamic allocation of CL point budget, or 0.95 of the total CL points budget, and divide the remaining budget equally amongst the IC/BC CL points. In the main paper, the presented results are for when the ratio is fixed to 0.8.
the experiments where fix the PDE CL points 0.5, 0.8
K
ADDITIONAL EXPERIMENTAL RESULTS
K.1 POINT SELECTION DYNAMICS OF PINNACLE
In this section, we further consider the dynamics of the point selection process between different algorithms. Figure 12] we plot the point selection dynamics of various point selection algorithms in the ID Advection equation problem. We can see that HAMMERSLEY will select points which best spans the overall space, however this is does not necessarily correspond to points which helps training the best. Meanwhile, RAD-ALL tends to focus its points on the non-smooth stripes of the solution, which corresponds regions of higher residual. While this allows RAD-ALL to gradually build up the stripes, it does so in a less efficient manner than PINNACLE. In the variants of PIN- NACLE, we can see that the selected CL points gradually build upwards. which allows the stripe to be learnt more efficiently. We can also see that PINNACLE-K selects points that are more diverse than PINNACLEE-S, which can be seen from the larger spread of CL. points throughout training, with PDE CL points even attempting to avoid the initial boundary in Step 200k.
(a) HAMMERSLEY
(b) RAD-ALL
3356
(c) PINNACLE-S
(d) PINNACLE-K
1000
Meg Sp000
220399
The TeEe
Mir Serec
233393
5051
PDE CL Pts
CL Pts
CL
Figure 12: Training point selection dynamics for PINNACLE in the ID Advection equation forward problem experiments.
we plot the point selection dynamics of various point selection algorithms in the ID Burger's equation problem. We can see that HAMMERSLEY peforms similar as in the Advection case, where it is able to span the space well but not able to guide model training well. Meanwhile, RAD-ALL selects points that are more spread out (that has high residual), but focuses less effort in regions near the initial boundary along the discontinuity in the solution, unlike PINNACLE. The points selection being less concentrated along these regions may explain why PINNACLE is able to achieve a better solution overall.
In Figure 13
30

---

Published as a conference paper at ICLR 2024
(a) HAMMERSLEY
(b) RAD-ALL
J000c0
Sep
Siep J00OCS
Low
(c) PINNACLE-S
(d) PINNACLE-K
Sry 1330
A30
0300
Figure 13: Training point selection dynamics for PINNACLE in the ID Burger's equation forward problem experiments.
ID Advection Equation, 200k steps
Uniform Rand
Hammersiey
RAD-AIl
PINNACLE-S (ours)
PINNACLE-K tours)
1.015
0.918
0.703
0,586
0,224
0.127
(b) ID Burger's Equation, 200k steps
Hommersiey
(ours)
0.8749
0.5959
0.3719
0.299
0.1500
0.0750
Figure 14: Predictive error for the forward problem experiments in Figure 4
K.2 FORWARD PROBLEMS
We plot the predictive error for experiments ran in Figure Figure Additionally, we plot the ID Advection and 1D Burger's equation experiments with additional point selection algorithms in Figures 13 and 16 respectively. Some observations can be made from these data. First, adaptive point selection methods outperform non-adaptive point selection methods. This is to be expected since adaptive methods allow for training points to be adjusted according to the current solution landscape, and respond better to the current training dynamics.
Second, for all the algorithms which do not perform dynamic CL point budget allocation, choosing whatever fixed budget at the beginning makes little difference in the training outcome. This can be seen that in each experiment, choosing the point allocation at 0.5, 0.8 or 0.95 shows little difference in performance, and is unable outperform PINNACLE. Meanwhile, the variants of PINNACLE are able to exploit the dynamic budgeting to prioritize the correct types of CL points in different stages of training.
31

---

Published as a conference paper at ICLR 2024
(a) Non-adaptive methods
(b) Adaptive methods
Mean errer
4 10
Mean error
x 10
10-1
2 10
100000 Steps
200000
100000 Steps
200000
Uni--33
Hamm-2.5
Sebol-3.5
Sobal-a.
01-0.5
Figure 15: Further experimental results for ID Advection equation forward problem.
(a) Non-adaptive methods
(b) Adaptive methods
Mean erro
Mean error
10
10
10-2
100000 Steps
200000
0
100000
200000
Steps
Unit..5
Hamm 0.5
Sobol Sobal-a.8
RAD-0.5
RAD A0--A1-555
FINNGLES FRNACLEK
Unito.s
Hamm-0.8
RAD-0.8 8.8
RAD-A1-30
UNIT-035
Hamm-0.95
S.00l.0.05
R.D
RAD A1--91-
Figure 16: Further experimental results for ID Burger's equation forward problem.
Third, while PINNACLE-K outperforms PINNACLE-S, it is often not by a large margin. This is an interesting result, since it may suggest that selecting a diverse of points may not be required for training PINN well.
Additionally, Figure 21a we also measure the runtime of different algorithms to train a 1D Ad- vection problem with 3 until it reaches mean loss of 10-2, We select this particular forward problem since it is known that Advection equation with a high training (Krishnapriyan et al. 2021). We can see that while some algorithms are able to converge with higher number of training points, they will end up also incurring a larger runtime due to the higher number of training points. Meanwhile. PINNACLE-E is able to achieve a low test error with much fewer points than pscudo-random sampling at a much quicker pace. This shows that to train PINNs with more CL points, this often having to compute the gradient of more training data. PINNACLE provides a good way to reduce the number of training points needed for training, which reduces this training cost problem.
difficult problem for PINN
while is possible
with computation cost of
In Figure 18 we also ran the point selection for the forward ID Diffusion equation where we apply hard constraint the PINN. This means the algorithm only requires selecting the PDE collocation point only. In the plots, we see that PINNACLE is able to still able to achieve better prediction loss than other methods, which shows that PINNACLE is still useful in collocation point selection even when the different training point types does not have to be considered.
K.3 INVERSE PROBLEMS
In Figure 19
we present further experimental results for the Navier-Stokes equation experiments.
We see that for the unseen components of the PDE solution (i.e., p, ê₁ and X2). PINNACLE is able to get much lower error values than the other algorithms (with exception of PINNACLE-K which is unable recover p as accurately). This shows that the point selection method is able to
32

---

Published as a conference paper at ICLR 2024
(a) ID Korteweg-oo de Vries Equation, 100k steps
Unifomm
RAD
2.164
Moah ctror
10-5
2.725
50002
(b) 2D Shallow Water Equation, 100k steps
167
22328
B3284
Woon Woinero
20/5/
50000
100000
Shiffrm Rand
Figure 17: Results from additional forward problem experiments. In each row, we show the plot of mean prediction error for each method, and the PINN output from different point selection methods. For Figure 17b the right-hand plots show the error of h(,, y, t) for 1.
Trus solution
Unilorm Rand
Hammersley
RADJAI
PEVACLE lours MANACLE- (ours)
8645
Wian cnor
0431
8.000
10-1
100.00
Staps
20000
30,00
1.15
1-2
1.-
22.24
Uniform Rand
Hammersiey
RAD AlI
PRINACLE S (ours)
PINNACLE K (ours)
Figure 18: Results from the ID Diffusion equation with hard-constrained PINN. The left-hand plot shows the mean prediction eиror for each method, while the right-hand hand plot shows the true value of the function and the predicted values from the PINN in the best trial.
select points to obtain the most information. For the prediction alone, all algorithms are able to perform about the same since they able to get the information from the experimental points and interpolate accordingly. For the unknown quantities, however, a simple point selection mechanism fails whereas PINNACLE is able perform better.
We also conduct additional inverse problem experiments on the Eikonal problem, as demonstrated in Figure 20 Unlike the Navier-Stokes problem whose aim learn the unknown constants of the PDE, the Eikonal inverse problem involves learning an unknown function v(2, y, 2) given the ob- served experimental readings T(x M, 2). We see that PINNACLE is able to better recover v(x, y, z) compared to the other point selection methods.
Furthermore, in Figure 21c we measure the running time for the Navier-Stoke inverse problem where we aim to compute the PDE constants for the Navier-Stokes equation, and measure the time required until the estimated PDE constants are within 5% of the true value. We can see that using 10x fewer points, PINNACLE-K outperforms the pseudo-random benchmark by being able to achieve the correct answer at a faster rate. Furthermore, since PINNACLE-K is able to converge with fewer steps, it means that in reality it would be much more efficient also since it requires less query for the actual solution, which saves experimental costs.
K.4 TRANSFER LEARNING OF PINNS
In Figure 2 we compare the point selection dynamics for each point selection algorithms for the experiments from Figure 6 We see that for PINNACLE, the CL. points selected in such
33

---

Published as a conference paper at ICLR 2024
(a) Prediction
(b) p(æ, v,
(c) A1
(d) Az
10"
Mon of
Moan erne
10-1
30-1
10-
10
30-1
10
10-1
501500
100000
50000 Staps
100000
50020 Steps
100000
50030
102000
Staps
Steps
Unitorm Rand
Hammersiey
RAI
RADAI
PINNACLE-S (ours)
PINNACLE-A (ours)
Figure 19: Errors of predicted value from the Navier-Stokes equation experiments. The graphs show, from left to right, the errors of overall prediction, the pressure p, A1 and A2.
Nore)
1902a
L003
Moen ener
0,02
6581
0.83
6143
-3300
- Linitorm Rand
- RAD
RAD-N
P PRRIACIESS (OUPN)
FARACLEE Jouns!
Figure 20: Results from the 3D Eikonal equation inverse problem. The left-hand plot shows the mean prediction enror for each method, while the right-hand plot shows the true value of v(r, z) and the PINN output for learned vxx, y; from different point selection methods when 1.
way that the existing solution structure still remains throughout training. This makes PINNACLE more efficient for transfer learning since it does not have to relearn the solution again from scratch. This is in contrast with the other methods, where the stripes eventually gets unlearned which makes the training inefficient.
In Figure 23 present more results from the transfer learning experiments for the Burger's 'ss equa- tion. As we can see, again, the variants PINNACLE are able to better reconstruct the solution with the alternate IC than other methods.
34

---

Published as a conference paper at ICLR 2024
(a) ID Advection (Forward)
(b) 3D Eikonal (Inverse)
107
0010
Hare 1346)
Harmn
HaTEL CLO0K
Hamm EUAC
BAC-B
2x
60
120
1.60
120
150
Time tr
Time (RE
(c) 2D Navier-Stoker (Inverse)
PHNNACEES IM
TANALLEKIN 13M
PINAACLE (11
3.00
Time 1r
Figure 21: Experiments involving runtime of various point selection algorithms. The highlighted lines in each plot represent the example which is able achieve the required accuracy and sustain the accuracy value for 5k additional steps within the shortest time. In Figures2laaand lines represent the threshold considered for acceptable accuracy for Advection equation problem, this marks the 10-2 accuracy point, while for the Navier-Stokes inverse problem, this marks the region where the estimated constant are within 5% of the true value.
the dotted
(a) HAMMERSLEY
(b) RAD-ALL
Stap
10000
Skap 50000
Shpp 200303
KI7I
X30
5001
02
130x
(c) PINNACLE-S
(d) PINNACLE-K
L35T
230333
KTTS
50M
PDE
Figure 22: Training point selection dynamics for PINNACLE in the ID Advection equation transfer learning problem experiments.
35

---

Published as a conference paper at ICLR 2024
Inital
Hemmersdey
RAD
PANNCLES PINACCE
2403
Roon cour
6.003
2335
Steps
Uniform Rand
Ho Nemmerssyy
--- RAD
R4D Al
+
PINIACEE (oUS))
PARACLEK Tours)
Figure 23: Predictive error for the ID Burger's equation transfer learning problem with perturbed IC.
36


Reviews:
[{"review_id": "6Qy42QWgVq", "reviewer": "Reviewer_hySh", "summary": "Joint selection of experimental points and collocation points in PINN training is proposed in this paper to achieve better training results for PINNs. A point selection algorithm that aims to select a training set that maximizes the convergence degree is proposed, which can lead to lower generalization bounds.", "questions": "1. In Algorithm 1, how to select SAMPLING or K-Means sampling strategy?\n2. Is the calculation of NTK used to compute the convergence degree?\n3.The runtime comparison is not provided.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper contains a substantial amount of mathematical proofs. The experimental results are excellent, and the algorithm exhibits significantly higher accuracy compared to other algorithms.", "weaknesses": "The algorithm seems straightforward, and the presentation can be more concise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AG70xUCrLw", "reviewer": "Reviewer_Eh91", "summary": "This paper proposes the PINNACLE, which is based on PINN network to address PDE problem. Specifically, proposed algorithm jointly optimizes the selection of all training points type and the authors demonstrate theoretically that criterion used by PINNACLE is related to generalization error of PINN.", "questions": "Check the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical results are interesting. The author connects the proposed convergence degree notion with the generalization error bound.  and demonstrates how to approximate the optimal set for convergence degree.\n\n2. The experimental results are great. Compared with other baselines, proposed methods introduce observable improvements.", "weaknesses": "1. For the K-MEANS++ method, the authors are encouraged to provide the time consumption comparison with other baselines. I am not sure if K-MENAS++ will introduce much extra computation cost.\n\n2. For the K-MEANS++ method, the author claims that \"this method select points with high convergence degrees\". The authors are expected to provide more explanation why this method increase the convergence degree. It is the same for SAMPLING method. The authors are expected to explain how to \"select a point which is proportional to $\\hat{\\alpha}(z)$ and how it improves the convergence degree\" explicitly.  \n\n3. For the experiments, the authors are encouraged to provide the results mean error with running time rather than only steps. Furthermore, since PINNACLE achieves great results in 1D Advection and 1D Burger settings, it will be interesting to explore with more settings, e.g., 1D Wave equation and 1D KdV equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBgHvlhcJA", "reviewer": "Reviewer_WCRx", "summary": "This paper proposes a method for adaptively selecting collocation points, which is necessary for training physics-informed neural networks. In particular, by treating the points related to experiments, partial-differential-equation constraints, and initial-boundary-value constraints in a unified manner, the proposed method automatically adjusts what proportion of collocation points should be allocated to each of them. The proposed method is based on a newly introduced criterion which is derived by using the theory of neural tangent kernels. The relationship between this criterion and generalization error has also been shown.", "questions": "As mentioned above, for the initial boundary values, a method of designing neural networks to satisfy them has been proposed, e.g. in the following paper. \n\nLagaris, I. E., Likas, A., and Fotiadis, D. I. (1998). Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000.\n\nWhat is the significance of the proposed method when such a method is employed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The choice of collocation points for PINNs is an important issue that has a significant impact on performance. In particular, this paper deals with collocation points in a unified way, independent of the type of loss functions associated with them. As far as I know, this is certainly a new approach and seems to be very promising.\n\nIn addition, the newly introduced criterion is derived with a theoretical basis and is highly reliable. This is just my impression but I suppose that the theorem that shows the relationship with the generalization error is itself valuable.", "weaknesses": "The strength of this paper seems to be that collocation points for initial boundary values and those for PDEs can be treated in a unified way, but a method of training networks without collocation points for initial boundary values is also proposed. When such a method is employed, the proposed method may lose a certain extent of significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bFQjrJc2Yr", "reviewer": "Reviewer_nqGc", "summary": "This work looks into adaptive experimental design for physics informed neural networks (PINNs) incorporating PDEs as constraints. The authors evaluate the design points from various types including experiments and initial and boundary conditions by computational constraint. They are then put together into the proposed PINN Adaptive Collocation and Experimental points selection, or PINNACLE, framework, which considers the interactions between the type of points selected. The authors theoretically demonstrate the relation of the framework to the generalization error, and the performance of PINNACLE compared to other design methods.", "questions": "1. My understanding is that the constraints by initial conditions or boundary conditions are only considered as \"soft\" constraints here due to how the objective function is set up. I guess for most situations its impact on the results would not be significant, or in other words, the approximation is going to be quite good. However, I wonder whether there could be cases where even a minor violation of the limiting conditions will cause significant distruptions to the system, and if so, how the authors propose to tackle that.\n2. This follows my previous question. I see the limiting conditions are introduced as regularization components into the objective function, with the corresponding regularizer coefficients described as having a certain value. I wonder whether the authors have considered how to best place values on these regularizers according to the physical system, and whether that would have a substantial influence on the final result. I presume different limiting conditions will have varying impact on the system, so I am curious on whether the selection of coefficients may be incorporated into the framework.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experimental design for physics-informed neural networks (PINN) is a very interesting problem, and incorporating the PDE constraints and the initial and boundary conditions into the system makes the objective of the work quite attractive and carry substantial weight.\n2. The notion of augmentation of points from various sources with the computational budget constraint is a unique idea that combines domain knowledge with experimental design strategies, which I think is of value.\n3. The work expressed the contribution and introduced the proposed framework clearly with illustrations that are quite helpful for understanding.", "weaknesses": "1. Not necessarily a weakness, but I am quite interested in the selection procedure of $Z_{pool}$ from $Z$ as introduced in Algorithm 1 and whether the authors have considered some structured designs that may give it more advantage than random sampling.\n2. Similar question regarding the selection procedure from subset $Z$ of $Z_{pool}$. I see two different design approaches are used here, but I wonder whether the authors have considered additional options, say weighted space-filling designs. Just a thought.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)