Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks
Authors: Woojin Cho
Kookjin Lee
Donsub Rim
Noseong Park
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Scientific machine learning
Physics-informed neural networks
Meta learning
Hypernetworks

Abstract: In various engineering and applied science applications, repetitive numerical simulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest a path that potentially opens up a possibility for physics-informed neural networks (PINNs), emerging deep-learning-based solvers, to be considered as one such solver. Although PINNs have pioneered a proper integration of deep-learning and scientific computing, they require repetitive time-consuming training of neural networks, which is not suitable for many-query scenarios. To address this issue, we propose a lightweight low-rank PINNs containing only hundreds of model parameters and an associated hypernetwork-based meta-learning algorithm, which allows efficient approximation of solutions of PDEs for varying ranges of PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as "failure modes" of PINNs.

Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks
Woojin Cho
Kookjin Lee
Donsub Rim
Noseong Park
Yonsei University
Arizona State University
Washington University in St. Louis
snowmoon@yonsei. ac.kr, kr, lee.jjinneeeeuuueeeemmm edu,
rim@wustl.edi noseong@yonsei.c. ac kr
Abstract
In various engineering and applied science applications, repetitive numerical sim- ulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest path that potentially opens up possibility for physics-informed neural networks (PINNs), emerging deep-learning- based solvers, to be considered as one such solver. Al- though PINNs have pioneered proper integration of deep-learning and scientific computing, they require repetitive tume-consumin which not suitable for many-query scenarios. To address this issue, we propose lightweight low-rank PINNs containing only hundreds of model parameters and associated hypernetwork-be
training of neural networks,
meta-learning algorithm, which allow efficient solution approximations for varying PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as "failure modes" of PINNs.
Introduction
Physics-informee neural networks (PINNs) [ are particular class of coordinate-basse multi-layer perceptrons (MLPs), also known as implicit neural representations (INRs), to numerically approx- imate solutions of partial differential equations (PDEs). That is, PINNs are taking spatiotemporal coordinates (x, t) an input and predict PDE solutions evaluated at the coordinates the (z. and are trained by minimizing (implicit) PDE residual loss and data matching loss at initial and bound- ary conditions. PINNs have been successfully applied to many different important applications in computational science and engineering domain: computational fluid dynamics [2] K. cardiac electrophysiology simulation material science [5], and photonics [3) to name a few.
PINNs are, however, sharing the same weakness with coordinate-based MLPs (or INRs), which hinders the application of PINNs/INRs more diverse applications; for new data instance (e.g.. new PDE for PINNs or new image for INRs), training new neural network (typically from scratch) is required. Thus, using PINNs to solve PDEs (particularly, parameterized PDE settings) usually computationally demanding, and this burden precludes the application of PINNs to important scenarios that involve many queries nature as these scenarios require the parameterized PDE models to be simulated thousands of times (e.g., design optimization, uncertainty propagation), i.e., requiring PDE solutions u(#, 1; (1) many PDE parameter settings (t) N₂ with very large N,..
Co-corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To mitigate the above described issue, we propose i) a low-rank structured neural network architecture for PINNs, denoted as low-rank PINNs (LR-PINNNS, ii) an efficient rank-revealing training algorithm. which adaptively adjust ranks of L.R -R-PINNs for varying PDE inputs, and iii) a two-phase procedure (offline training/online testing) for handling many-query scenarios. This study inspired by the observations from the studies of numerical PDE solvers 7+11 stating that numerical solutions of parametric PDEs can be often approximated in a low-rank matrix or tensor format with reduced computational//eeoor requirements. In particular, the proposed approach adopts the computational formalism used in reduced-order modeling (ROM) 12] [ one of the most dominant approaches in solving parameteric PDEs, which we will further elaborate in Section
In essence, LR- LR-PINNS represent the weight of some internal layers as low-rank matrix format. Specifically, we employ singular- wnggulla-valueddecompsstt decompositionn (SVD)-like matrix decomposition, i.c. a linear combination of rank-1 matrices: the weight of the 1-th layer is W E s with the
rank r = min(ng, m(+1), where W E Rm411x1, ut E R vf E RR" and s R. The ranks of the internal layers, however, typically are not known a priori. To address this issue, we devise a novel hypernetwork-
neural network architecture, where the rank-structure depending on the PDE parameters is learned via training. In short, the proposed architecture consists of i) a lighteeight hypernetwork module and ii) a low-rank solution network module; the hypernetwork takes in the PDE parameters and produces the coefficients of the rank-1 series expansion (i.c., s(4) f (u)). The low-rank solution network module i) takes in the spatiotemporal coordinates (x. t). ii) takes the forward pass through the linear layers with the weights (p) = Σ.nn s ( T of which (() comes from the hypernetwork, and iii) produces the prediction uh (xt t; u)). Then, the training is performed via minimizing the PINN loss, i.e., a part of the PINN loss is the PDE residual loss, [[ (I, 1; 11) where R(:;;() (4) denotes the parameterized PDE residual operator.
We show the efficacy and the efficiency of our proposed method for solving some of the most fundamental parameterized PDEs, called sinrecctoondiffuiion-- reaction equations, and Helmholtz equations. Our contributions include:
1. We employ a low-rank neural network for PINNs after identifying three research challenges to address.
2. We develop a hypernetwork-based framework for solving parameterized PDEs, which computes solutions in rank-adaptive way for varying PDE parameters.
3. We demonstrate that the proposed method resolves the "failure modes" of PINNs.
4. We also demonstrate that our method outperforms baselines in terms of accuracy and speed.
2 Naïve low-rank PINNs
Let us being by formally defining LR-PINNs and attempt to answer relevant research questions. LR-PINNs are a class of PINNs that has hidden fully-comnected layers (FC) represented as a low-rank weight matrix. We denote this intermediate layer LR--FC: : the 1-th hidden layer is defined such that
h 1+1 (h'))
h |+1 (+ (V, h²)) +b',
(1)
where U. E R and Vi R denote full column-rank matrices (i.e., rank r E n1 RI+1) containing a set of orthogonal basis vectors, and Σ E BRrxr is a diagonal matrix, ΣS = diag (s:)) with sf E R
LR-PINNs with a rank of r and L hidden layers require O((2n) + 1)rL) memory as opposed to 0 L) required by regular PINNs.
Memory efficiency:
The forward/backward pass of LR-PIINN can be computed efficiently by utilizing a factored representation of the weights. To simplify the presentation, we describe only the forward pass computation; the forward pass equivalent to perform three small matrix-vector products (MVPs) in sequence as indicated by the parentheses in Eq.
Computational efficiency:
Representing the weights of hidden layers itself straightforward and indeed has been studied actively in many different fields of deep learning, e.g., NLP 14 [5] However, those approaches typically assume that there exist pre-trained models and approximate the model weights by running the truncated SVD algorithm. Our approach is different from these approaches in that we
Challenges:

---

playpr (14)
PDE parameters
Paramotarized PDBs
Hiag(="(w)) (44)
diag( (1))
diag( (1))
Conrdinates
Linear layer
(o)
t
(x)
Emnor layer
Aaa
A
A
h-r
(x+
uu ue, t))
1
yr
LR-FOA
LR-FC'
IR-FC
Figure 1: The architecture of Hyper-LR- LRper-LR-PINN consisting of i) the hypernetwork generating model parameters (i.e., diagonal elements) of LR-PINN and ii) LR-PINN approximating solutions.
attempt to reveal the ranks of internal layers as the training proceeds, which brings unique challenges. These challenges can be summarized with some research questions, which include: C1) "should we make all parameters learnable (i.e., (U', VL, 2'))?" C2) "how can determine the ranks of each layer separately, and also adaptively for varying j!?", and C3) "can we utilize a low-rank structure to avoid expensive and repetitive training of PINNs for every single new μi instances?". In the following, we address these questions by proposing a novel neural network architecture.
3 Hyper-LR-PINNS: hypernetwork-based meta-learning low-rank PINNs
We propose a novel neural network architecture based on a hyperneworr and an associated training algorithm, which address the predescribed challenges. To distinguish them, we hereinafter call LR- PINN with (resp. w/o) our proposed hypernetwork "Hyper-LR-PINN" (resp. "Naive-LR-PINN"),
Design goals: Here we attempt to resolve challenges by setting up several design goals that are inspired from our domain knowledge in the field (and also from some preliminary results that can be found in Appendix D):
1. build single set of basis vectors U and v', preferably as orthogonal as possible, that perform well over a range PDE parameters,
2. build LR-PINNs with an adaptive and layer-wise rank structure that depends on PDE parameters (e.g., a higher rank for higher convective PDE) and,
3. make only the diagonal elements, denoted s in Eq. learnable to achieve high efficiency once proper set of basis vectors and the rank structure are identified.
Our design largely follows the principles of ROMs, where the expensive computation is offloaded offline phase to build cheap surrogate model that can perform efficient computation in an online phase for a test set. To make an analogy, consider parameterized dynamical systems (which may arise in semi-discretization of time-dependent PDEs): du(t:u) f(u(t; p)) where
dt
N In the offline phase, the method seeks low-dimensional linear trial basis where the reduced representation of the solution lie on, which is achieved by performing high-fidelity simulations on a set of PDE parameter instances {u (6) Tuain and constructing a trial linear subspace := range V pp) Jiml
Thain
with V [v) bpp E Nx from the solution snapshots collected from the high-fidelity simulations. In the online phase, the solutions at a set of test PDE parameter instances {μ(() the approximated utt, 14) Ct (1) with E IR,, and low-dimensional surrogate problem is derived as de(t:p)) = ( ((t: f(c(t: p): E TRP, which can be rapidly solved, while de not losing too much accuracy. See Appendix C for an illustrative explanation and details of ROMs.
Taking a cue from the ROM principles, we design our model operate on a common set of basis vectors } and [V,'), which are obtained during the offline phase (analogous to V in ROMs) and update only the diagonal elements {s!} during the online phase (analogous to e in ROMs). Now, we elaborate our network design and the associated two-phase algorithm. For the connection to the ROM, which explains details on the context/query sets, be found in Appendix
3

---

3.1 Hypernetwork-band baasd neural network architecture
The proposed framework has two computational paths: a path for the hypernetwork and path for LR-PINN (Figure D. The hypernetwork path reads the PDE parameter and outputs the diagonal elements (s 1s')1_ ]|=1 of LR-FCs. The LR-PINN path reads (z. t) and the output of the hypemetwork, and outputs the approximated solution uh at (æt, 1: 14), which can be written follows:
ue u :) = ue t); (
where (4) denotes the hypemetwork such that {=) = fhy (p). We denote s as a function of μa to make explicit that it is dependent on μ.. The internals of LR-PINN be described as with hD x t]
h = ( +b%)
h = (I) +b),, L,...,,....
we (ax t) o o h + bbbt)).
where (x) (w)) The hypernetorrk can be described as the following initial embedding layer, where e = μs. followed by an output layer:
Th (Wrmmb,m em b.mb...) m ..... M,
s'(x) = Reh + Blyper.)), 1=
where ReLU is employed to automatically truncate the negative values that the adaptive rank structure for varying PDE parameters can be revealed (i.e., the number non-zeros (NN7s) p in s (p)
varies depending on nt)
3.2 Two-phase training algorithm
Along with the framework, we present the proposed two-phase training algorithm. Phase 1 for learning the common set of basis vectors and the hypernetwork and Phase 2 is for fine-tuning the network for specific set test PDE parameters. Table shows the sets of model parameters that are being trained in each phase. (See Appendix E for the formal algorithm.)
In Phase we train the hypernetwork and the LR-PINN jointly on a set of collocation points that are collected for varying PDE parameters. Through the computational procedure described in Section 3.1 the approximated solutions at the collocation points are produced ue( ((z, tty (0)). Then, as regular PINNs, the PDE residual loss and the data matching loss can be computed. The small difference is that the PDE operator, F, for the residual loss is also parameterized such that Flue t2 ): ()); ()). As we wish to obtain basis vectors that are close to orthogonal, we add the following orthogonality constraint based on the Frobenius norm to the PINN loss [6]:
WillU'TT T /|| + u2 V'TV' T|
where wh and w2 are penalty weights. (See Appendix
In Phase 2, we continue training LR-PINN for approximating the solutions of a target PDE parameter configuration. We i) fix the weights, the biases, and the set of basis vectors of LR-FC obtained from Phase 1, ii) convert the diagonal elements to a set of learnable parameters after initializing them with the values from the hypernetwork, and iii) detach the hypernetwork. Thus, only the trainable parameters from this point are the set of diagonal elements, first and last linear layers. The hypernetwork initialized update optimizers (i.e., require less number of epochs). Moreover, significant computational savings can be achieved in the gradient update steps as only the diagonal elements are updated (i.e., 8141 6- + 1) V. J. instead of O,+1 <- θ + L)).
-nnttmetworkiitttaliieeddd diagonal elements serve as good starting point in (stochastic) gradient
Experiments
We demonstrate that our proposed method significantly outperforms baselines the 1-dimensional/2- dimensional PDE benchmarks that are known to be very challenging for PINNs to learn [8). We
report the average accuracy and refer readers Appendix for the std. dev. of accuracy after 3 runs.
'The NNZs is not equal to the rank the learned basis is only cofttconsta to be orthogonal (Eq. 2) Nevertheless, to avoid notation overloading, we denote NNZs the rank in the following.

---

Table 1: Learnable parameters in each phase
{(U", V 6¹) b')) Mi-1 W W D b 1(Wamb; b 1M
(LR-PINN),
Phase
Im {(Whypers b M (hypernetwork),
Phase 2
1o'1 w W ,
(LR-PINN)
75
75
75
25
25
00
K
100
30
0
50
022
0.A
0.5
0.8
1.0
00
0.2
0.4
6.6
0.8
1.0
00
80
02
0.A
0.5
0.8
1.0
0
(a) Exact solution
(b) PINN
(c) Ours
Figure 2: [Convection equation] Solution snapshots for 3 40
Baselines for comparison: Along with the vanilla PINN M. our baselines include several variants. PINN-R [9 denotes model that adds skip-connections PINN. PINN-S2S [8]] denotes method that uniformly segments the temporal domain and proceeds by training each segment one by one in a temporal order. PINN-P denotes method that directly extends PINNs take (x, t. (1) as input and infer ug (x, f, (4), i.e., 4l is being treated as coordinate in the parameter domain.
We also apply various meta-learning algorithms Naive-LR-PINN: model- model-agnostic meta learning (MAML) 20 and Reptile [21] recall that Naive-LR-PING means that LR-PINN without our meta-learning and therefore, MAML and Reptile top of Naive-LR- PINN can compared to Hyper-LI -eeee-LR-PNNNN. In the parameterized PDE setting, we can define a task, 7(), as a specific setting of the PDE parameters, f Both MAML and Reptile seek initial weights of PINN, which serve good starting point for gradient- based optimizers when solution of a new unseen PDE parameter setting is sought. See Appendix for details. For reproducibility, refer readers Appendix F including hyperparameter configuration and software/hardware environments.
proposed hypernetwork-base
Evaluation metries: Given the i-th PDE parameter instance (() the ground-truth solu- tion evaluated at the set of test collocation points can be defined collectively as ( [u(z1, 11 u((,), (N; and likewise for PINNs as u.. Then the absolute error and the relative error can be defined as 1 (l) u || and [|u(()) /
||2 (0) respectively. In
Appendix NJ we measure the performance on more metrie: max error and explained variance score.
4.1 Benchmark parameterized PDES
1D PDEs: For first benchmark parameterized PDEs, we consider the following parameterized cunvection-difusionnreeee (CDR) equation:
Wt + Bug W'la pu(1 w) 0, I E Ω L E [0,T],
where the equation describes how the state variable evolves under convective (the second term), diffusive (the third term), and reactive (the fourth term) phenomena The triplet, μ (3, v, p). defines the characteristics of the CDR equations: how strong convectieddifftssvveecaatt is, respectively. A particular choice of μ leads realization of specific type of CDR processes.
the equation
There a set of μ which makes training PINNs very challenging, known as "failure modes" 18 i)
convection equations with high convective terms (33 2 30) and ii) -diffioo-iif equations with high reactive terms (p 2 5). We demonstrate that our method does not require specific-PDD PDE-deddicated algorithms (e.g.. while producing comparable/better accuracy in low rank.
Note that consider the Fisher's form pu(1 u) the reaction term following
B

---

10
Repple
NAML
1,0
MAM Reptlle Dura
10
MAML Reptle
Our
relative error
Oura
0.
7
10-3
J0.5
2000
4000
60OD
B200
30000
2000
4000
6000
AoDO
10000
2000
6000
6000
8000
10600
Epoch
Epoch
Epoch
(a) Train loss
(b) The absolute error
(c) The relative error
Figure 3: [Convection equation] Per epoch averaged train loss and test errors for 8 E [30, 40] for Phase ameta-learning phase). We refer readers to Appendix SorendixOffor Phase 2 loss curve.
Table 2:
[Convection equation] The absolute and relative errors of the solutions of convection equations with / {30, 35, 40). We apply the curriculum learning proposed in [ MAML, and Reptile to Naïve- -R-PNNN Therefore, we focus on comparing various Naive-LR SELLR-PIN--OE based enhancements and our Hyper-LR-PINN, See Appendix N for other omitted tables.
[wo) Pre-truining
[w] Pre-training
8
Rank
Natve-I.R-PINN
Curriculum leurning
MAMI.
Reptile
Hyper-LR-PIn (Full runk)
Hyper-LR-Pee LL-PIN (Adaptive runk)
Abs.
Rel.
Ahs,
Rel.
Abs.
Rel. err.
Abs, err.
Rel.
Abs.
Rel.
Abs.
Rel.
0.5617
0.5344
0.4117
0:4098
0,6757
0.6294
0.5893
0.5551
0.5501
0.4023
0.6452
0.5779
30
0.5327
0.4233
0.4204
0.5781
0.5451
0.6048
0.5704
0.0360
0.0379
0.0375
0.0389
40
0.5257
0.5076
0.3746
0.3744
0.5848
0.5515
05757
0.5442
0.5327
0.5126
0,4152
0.4127
0.5898
0.5562
0.5817
0.5496
0.5663
0.5357
0.5825
0.5465
0.6663
0.6213
0.5786
0.5446
20
0.5675
0.5369
0.6120
0.5673
0.6814
0.6433
0.5971
0.5606
35
30
0.6081
0.5670
0.5864
0.5503
0,5819
0.5466
0.8866
0.5506
0.0428
0.0443
0.0448
0.0461
40
0,5477
0.5227
0.5954
0.5548
0,5809
0,5462
0.5773
0.5435
50
0,5449
0.5208
0.6010
0,5619
0,5870
0,5514
0.5731
0.5404
10
0,5974
0.5632
0.5978
0,5611
0,6789
0.6446
0.5992
0.5632
20
0,5890
0.5563
0.6274
0,5820
0,7008
0.6801
0.6189
0.5853
40
30
0.6142
0.5724
0.6011
0,5652
0.6072
0,5700
0.6126
0,5810
0.0603
0L0655
0.0656
0,0722
0.5560
0.5293
0.6126
0.5715
0.5832
0.6004
0.5638
0.6161
0.6130
0.5799
0.6007
0.5645
2D PDEs: equation:
As the second set of benchmarks, we consider the 2-dim parameterized Helmholtz
Uge Uyy k2u - q(I, yr, 01, = 0,
where q(2, ys 1 02) denotes specific parameterized forcing term and the solution u can be caleu- lated analytically (See Appendix D As observed in the failure modes of convection equations (high convective terms), certain choices of the parameters in the foreing term, i.e., @1, 02, make the training of PINNs challenging as the solutions become highly oscillatory.
4.2 Experimental results
4.2.1 Performance on the failure mode of the CDR equation
Solution accuracy: As studied in prior work, vanilla PINNs fail to approximate solutions exhibiting either highly oscillatory (due to high convection) or sharp transient (due to high reaction) behaviors. Here, we present the results of convection equations and leave the results on saactio(-ddiffuioon equations in Appendix N For convection equations failures typically occur with high 3, e.g., reports the results of all considered models trained on 3 E [30, 40 and essentially 3 2 30. Table 2 shows that Hyper-LR- -R-PINN (Figure e)) is the only low-rank method that can resolve the failure- mode and there only marginal decreases in accuracy compared to Hyper-LR-PINN in full rank. For reaction( reaction(-diffusionn (-diffusion) equations, we observe similar results, Hyper-LR-PPie outperforms the
$ 'nitial condition: + sin(x) and boundary condition: periodic FFllowing [18, we consider only the coefficients in the natural numbers (e.g., 9 E N4.) and [30,40] indicates the set (30, 31,...,40)) 40).

---

Table 3: Comparisons of model size
Model
Naive-L.R-Pi
Ours
PINN
Rank
10
20
30
40
50
Adaptive
# Parameters
381
411
441
471
501
<-51
10,401
Adaptive rank
32
34
19
10
L 30
8
28
28
-5 26
9-
2-24
22
27
20
30
32
34
36
40
10
13
16
15
22
29
36
43
50
0
(p4)
(a) 8 E [30,40]
(b) 9 E [1,20]
(c) B [1,20]
Figure 4: Adaptive rank on convection equation (the left and the middle panels). The magnitude of
the learned diagonal elements 8² of the second hidden layer for varying 33 E [1, 20] (the right panel).
best baseline by more than an order of magnitude (See Appendix N Moreover, we report that Hyper-LR-PINN outperforms in most cases to the baselines that operate in "full-rank": meta-learning
methods (MAML, Reptile), PINN-S2S, and PINN-P (Appendix M
Loss curves: Figure depicts two curves of train loss and test errors as the meta-lcarning algo- rithms proceed (MAML, Reptile, and ours). As opposed to the optimization-bann algorithms, which tend to learn meta-initial weights that perform well "on average" over randomly sampled training tasks, our hypernetwork-basee method minimizes the loss for each individual task simultaneously. Loss curves for Phase are reported Appendix O which essentially show that the baseline meta-learners do not provide good initializations.
meta-learning
Rank structure: Table B compares the number of trainable model parameters for Naive-LR- PINN, and our method. In our model, each hidden layer has different rank structure, leading to 351 trainable parameters, which about (x30) smaller than that of the vanilla PINN, Cf. merely decreasing the model size of PINNs leads to poor performance (Appendix SS.. Figure a shows how
model adaptively learns the rank structure for varying values of 3. We observe that, with low-rank format, Hyper-LR- PINN provides more than (x4) speed up in training time compared to the vanilla PINN. More comparisons on this aspect will be presented in the general CDR settings.
Ablation on fixed or learnable basis vectors: We compare the performance of the cases, where the basis vectors {UZ, V/} are trainable. We observe that the fixed basis vectors lead to an order of magnitude more accurate predictions as well as faster convergence (See Appendix Q for plots).
Ablation orthogonality constraint: We compare the performances of the cases, where the proposed model is trained without the orthogonality constraint, Eq. 2 Experimental results show that the orthogonality constraint essential in achieve good performance (See Appendix !].
Many-query settings (inter/extra-polation in the PDE parameter domain): Next, we test our algorithm on the many-query scenario, where we train the proposed framework with the 11 training PDE parameters [30, 40] (Phase 1), freeze the hypernetwork and the learned basis, and retrain only the diagonal elements for 150 unseen test PDE parameters (Phase 2). Figure 5 depicts the absolute and relative errors of the solutions, which are as accurate as the ones on the training PDE parameters, in particular, for 3 < 30 (i.e., extrapolation). Although the model produces an increasing error for 3 40, the relative error is still around ~00% which cannot be achievable by vanilla PINNs. The right chart of Figure shows the number of trainable model parameters, which is determined by the rank (i.e., the output of the hypernetwork). As PINNs fail to produce any reasonable approximation in this 3 regime, make comparisons in the general CDR settings below.
7

---

absolute error
0.44
Phoet Mase2
0.1
Phoni Mase
0.14
1.15
0.12
Model
0.00
0.05
0,00
5000
J0.4
(a) The absolute error
(b) The relative error
(c) Model size
Figure 5: -uery scenario: the absolute/relative errors of the predicted solutions at unseen test parameters (left and middle), and the number of trainable model parameters (right).
Table 4: The relative errors of the solutions of parameterized PDEs with 8, E [1,5]. The initial condition of each equation is Gaussian distribution N N(m, (T/2)²).
No pre-training
Meta-learning
PDE type
PINN
PINN-P
PINN-S2S
MAML
Reptile
Ours
Convection
0.0327
0.0217
0.2160
0.1036
0.0347
0.0085
Reaction
0.3907
0.2024
0.5907
0.0057
0.0064
0.0045
Conv-Diff-Reat
0.2210
0.2308
0,5983
0.0144
0.0701
0.0329
4.2.2 Performance on the general case of the CDR equation
Next, we present the performance of the proposed method on solving the benchmark PDEs (not only in the failure mode but) in all available settings for B, v. p. Due to space reasons, we present only a subset of our results and leave the detailed results in Appendix M Table shows that the proposed Myyee-LLR-PINE mostly outperforms the baselines that operate in full rank.
Solution accuracy:
Some observations on learned low-rank structures: Again, the proposed method learns rank structures that are adaptive to the PDE parameter (Figure ()). The learned rank structures vary more dynamically from 20 34) as the PDE parameter range [1,20] larger compared to [30,40].
Figure c) visualizes the magnitude of the learned diagonal elements s² in the second hidden layer. The horizontal axis indicates the index of the element in s², which is sorted in descending order for analysis, and the vertical axis indicates the 3 value. The depicted result agrees with our intuitions in many aspects: i) the values decay fast (indicating there exist low-rank structures), ii) the value of each element s either increases or decreases gradually as we vary 3, and iii) for higher 3, higher frequency basis vectors required capture more oscillatory behavior exhibited in the solutions, which leads to higher ranks. We report the information for other layers and other problems in Appendix L
Computational cost: Here, we compare the computational cost of Hyper-LR-PINN and PINN in many-query scenario. We train Hyper- LR-PINN for 3 E [1, 20] with interval (Phase 1) and perform Phase
MBh typaatLi--
8000
on E [1, 20) with interval 0.5 (including unseen test PDE parameters). We record the number of epochs required for PINN and Hyper-LR-PINE reach an I.2 absolute error of less than 0.05. Figure 6 reports the number of required epochs for each 3, showing that the number of epochs required for Hyper-LR- increases rapidly: for most cases, Hyper-LR-PING reaches the target (i.c., 0.05) in only one epoch. We also emphasize that Hyper-LR-PINN in Phase requires much less FLOPS in each epoch (See Section 3.2
6000
2000
slowly increases while that for PINNs
5
10
15
20
Figure 6: Computational cost in epochs
4.2.3 Performance on the 2D Helmholtz equation
Now report experimental results on 2D Helmholtz equations for varying a and az Under the same condition in Table 2 we train Hyper-I.R-PINN for a E [2. 3] with interval 0.1 (a = a1 a2). and compare the performance with PINN. Surprisingly, Hyper-LR LR-PINN approximates the solution
8

---

1.6
100
95
175
675
925
225
225
225
121
122
275
2.73
6.73
1..0
-0.5
0.0
1.0
1,.0
-6.5
0.5
1.0
1,1,0
-0.
05
x
(a) PINN (Abs.em::0074003
(b) Ours (Abs.er.:000088))
(c) Exact solution
Figure 7: [2D-Helmholtz equation] Solution snapshots for a = 2.5
with high precision in only 10 epochs in Phase 2, while PINN struggles to find accurate solutions
over 2,000 epochs (Figure We show more visualizations and detailed results in Appendix
Related Work
Meta-learning of PINNS and INRs: HyperPINNs
share commonalities with pro- posed method in that they generate model parameters of PINNs via hypernetwork [3] However, hyperPINNs can only generate full-rank weights and do have capabilities to handle parameter- ized PDEs. Another relevant work is optimization-baase meta-learning algorithms; representatively MAML 200 and Reptile [2]] In the area of INRs, meta-learning methods via MAML and Reptile for obtaining initial weights for INRs have been studied in [24]. In [25] 2 meta- meta-learning methods, which are based on MAML,, have been further extended obtain sparse representation INRs.
Low-rank formats in neural networks: In natural language processing, the models being em- ployed (e.g., Bert) typically have hundreds of millions of model parameters and, thus, making computation efficient during the inference is of the imminent issues. As remedy, approximating layers low-rank via truncated SVD has been studied [EL [5] Modeling layers in low-rank in general has been studied for MLPs 27 R0 and convolutional neural network architectures [II In the context PINNs or INRs, there no low-rank format has been investigated. The work that is the closest to work is SVD-PINNs Eq. I but always in full-rank.
32 which represents the hidden layers in the factored form in
PINNs and their variants: There have been numerous sequels improving PINNs in many different aspects. One of the main issues is the multi-term objectives; in [19]. special gradient- based optimizer that balances multiple objective terms has been proposed, and in [33, a network architecture that enforce boundary conditions by design has been studied. Another main issue is that training PINNs often fails on certain classes of PDEs (e.g., fail to capture sharp transition in solutions over the spatial/temporal domains). This is due to the spectral bias [34] and, as remedy, in [ training approaches that gradually increase the level of difficulties training PINNs have been proposed. In k51 a Lagrangian-type reformulation of PINNs is proposed for convection-dominate PDEs. There are also other types of improvements including Bayesian-
PINNs 36 and PINNs that enforce conservation laws [37]] All these methods, however, require training from scratch when the solution a new PDE is sought.
Conclusion
In this paper, we propose low-rank formatted physics-informed neural networks (PINNs) and hypernetwork-basee meta-learning algorithm to solve parameterized partial differential equations (PDEs). Our two-phase method learns a common set of basis vectors and adaptive rank structure for varying PDE parameters in Phase and approximate the solutions for unseen PDE parameters by updating only the coefficients of the learned basis vectors Phase 2. From the extensive numerical experiments, we have demonstrated that the proposed method outperforms baselines in terms of accuracy and computationaa/meemrry efficiency and does not suffer from the failure modes for various parameterized PDEs.

---

Limitations: Our Hyper-LR-PINN, primarily concentrates on the cases where the PDE parameters are the coefficients of PDE terms. As future study, we plan to extend our framework to more general settings, where the PDE parameters define initial/boundary conditions, or the shape of domains. See Appendix for more details.
Broader impacts: HlyPerLRR-PINN can solve many equations even for PINN's failure modes with high precision. However, despite its small errors, one should exercise caution when applying it to real-world critical applications.
Acknowledgments
This work was supported by the Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Arti- ficial Intelligence Graduate School Program ay Yonsei University, 10%). and (No.2022-0-00857, Development of Aldattaasee finacial/ccooomic digital twin platform, 90%).
References
[1] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed
informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[2] Zhiping Mao, Ameya Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 1 12789, 2020.
[3] XIA Yang, Suhaib Zafar, J-X Wang, and Heng Xiao. Predictive large-eddy-simulatio wall modeling via ahysics-informed neural networks. Physical Review Fluids, 4(3):034602, 2019.
[4] Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
[5] Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using physics-inforse informed neural networks. Science advances, 8(7)xcabk0644, 2022.
[6] Wei Ma, Zhaocheng Liu, Zhaxylyk A Kudyshev, Alexandra Boltasseva, Wenshan Cai, and Yongmin Liu. Deep learning for the design of photonic structures. Nature Photonics, 15(2):77- 90, 2021.
[7] Daniel Kressner and Christine Tobler. Low-rank tensor Krylov subspace methods for parametrized linear systems. SIAM Journal on Matrix Analysis and Applications, 1888): 1316, 2011.
[8] Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. GAMM-Mitteillngen 36((1):3377, 2013.
[9] Kookjin Lee and Howard C Elman. A preconditioned low-rank projection method with rank-reduction scheme for stochastic partial differential equations. SIAM Journal on Scientific Computing, 39(5):8828-$850 2017.
[10] Markus Bachmayr, Albert Cohen, and Wolfgang Dahmen. Parametric PDEs: Sparse or low-rank approximations? IMA Journal of Numerical Analysis, 1661)1166111708 2018.
[11] Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems nonlinear mani- folds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
[12] Philip Holmes, John Lumley, Gahl Berkooz, and Clarence W Rowley. Turbulence, coherent structures, dynamical systems and symmetrry. Cambridge university press, 2012.
10

---

[13] Peter Benner, Serkan Gugercin, and Karen Willcox. A survey of projection-based model reduction methods for parametric dynamical systems. SIAM review, 57(4):483-331 2015.
[14] Patrick Chen, Hs Yu, Inderjit Dhillon, and Cho-Jui Hsich. Drone: Data-aware low- rank compression for large NLP models. Advances in neural information processing systems, 34:29321-29334, 2021.
[15] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[16] Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 678-679, 2020.
[17] Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-ifformed neural networks using a soft attention mechanism. arXiv preprint crXiv:2009.04555 2020.
[18] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in telyssics-nffommee neural networks. Advances in Neural Information Processing Systems, 34:26548-26635 2021.
[19] Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin, and Noseong Park. Dpm: A novel training method for physics-informed neural networks extrapolation, In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8146-8154, 2021.
[20] Chelsea Finn, Pieter Abbeel, and Sergey Levine. agnostig meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. 1135. PMLR, 2017.
[21] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint /803.020000999 2018
[22] Filipe de Avila Belbute-Peres, Yi-fan Chen, and Fei Sha. HyperPINN: Learning parameterized differential equations with physics-informee hypernetorks.. In The Symbiosis of Deep Learning and Differential Equations, 2021.
[23] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 16X9:01600900006 2016.
[24] Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural repre- sentations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2846-2855, 2021.
[25] Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34:1 34:11769-1119 11780, 2021.
[26] Jonathan Richard Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. arXiv preprint arXiv:2205.0880 088557, 2022."
[27] Tara N Sainath, Brian Kingshury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6655-6659, IEEE, 2013.
[28] Yu Zhang, Ekapol Chuangsuwanich. and James Glass. Extracting deep neural network bottle- neck features using low-rank matrix factorization. In 2014 IEEE international conference on acoustics, speech and signal processing, pages 185-189. IEEE, 2014.
[29] Yong Zhao, Jinyu Li, and Yifan Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5005-5009. IEEE, 2016.

---

[30] Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regular- ization of factorized neural layers. International Conference on Learning Representations. 2020.
[31] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
[32] Yihang Gao, Ka Chun Cheung, and Michael K Ng. SVD-PINNS: Transfer learning of physics- informed neural networks via singular value dccomposition. arXiv preprint arXiv:221..08760 2022.
[33] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing. 43(5):A3055-A308) 2021.
[34] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[35] Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Kolmogorov n-width and Lagrangian physics-informed neural networks: A causality-coffrriing manifold for convection- dominated pdes. Computer Methods in Applied Mechanics and Engineering, 404:1 155100 2023.
[36] Liu Yang, Xuhui Meng, and George Em Karniadakis. B-PINNs:
Bayesian physics-informen neu- ral networks for forward and inverse PDE problems with noisy data. Journal of Computational Physics, 425:109913, 2021.
[37] Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics physics-informed informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365: 1 13028. 2020.
[38] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
[39] Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 665-674, 2013.
[40] Alireza Doostan and Gianluca Iaccarino. A least-squares approximation of partial differ- ential equations with high-dimensional random inputs. Journal of computational physics, 228(12) 4345, 2009.
[41] Sergey V Dolgov and Dmitry V Savostyanov. Alternating minimal energy methods for linear systems in higher dimensions. SIAM Journal on Scientific Computing, 36(5) 2014.
[42] Kookjin Lee, Howard C Elman, Catherine E Powell, and Dongeun Lee. Enhanced alternat- ing energy minimization methods for stochastic galerkin matrix equations. BIT Numerical Mathematics, 62(3))965-994, 2022.
[43] K. Dingle, C. Q. Camargo, and A. A. Louis. Input-output maps strongly biased towards simple outputs. Nature communications, 2018.
[44] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the carly-time learning dynamics of neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
[45] Minyoung Huh, Hossein Mobahi, Richard Zhang, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv, 2021.
[46] Benjamin Schrauwen, David Verstraeten, and Jan Van Campenhout. An overview of reservoir computing: theory, applications and implementations. Proceedings of the 15th european symposium on artificial neural networks. p. 471-482 2007, pages 47 471-482, 2007.
12

---

[47] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-leve performance on ImageNet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[48] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
13


Reviews:
[{"review_id": "5TANN2kjO7", "reviewer": "Reviewer_W4Fc", "summary": "The paper proposes an approach for solving PDEs by using a low-rank parameterized NN.\nWhile traditional PINNs are restricted to a single PDE instance, the authors use a hypernetwork to be able to solve PDEs with different parameters. The hypernetwork predicts the weights matrices of a PINN network in a low-rank fashion: each weight matrix is decomposed via SVD, and only the singular values diagonal matrix are modulated by the hypernetwork, while the other two matrices are shared across all the different PDEs. This allows quick inference on unseen PDEs, by learning only the diagonal terms.\nPerformance is evaluated on 1D-convection-reaction-diffusion and 2D-Helmholtz PDEs. It is shown that this low-rank hypernetwork architecture allows to learn on previous failure cases of the vanilla PINN (eg high convection or high reaction). \nThe hypernetwork method shows strong performance, compared to other meta-learning methods (MAML, Reptile), as well as other PINN variants.", "questions": "- \"For higher values (20), it appears that a higher rank is required to achieve a certain level of accuracy, presumably due to numerical reasons.” (l.503): what do you mean by numerical reasons?\n- How does the proposed architecture scale to higher dimensions?\n- How does the Phase 1 computation compare to the PINN variants?", "limitations": "The method is currently limited to generalizing to new PDE parameters only, it would be interesting to see if it can be extended to different initial / boundary conditions", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, and provides an extensive set of experiments and ablations showing the relevance of the low-rank decomposition, proposed loss terms and evaluate\n- The performance of the proposed model is very competitive, while also providing much faster inference compared to the other meta-learning algorithms, and generalizing better than the other PINN variants", "weaknesses": "- “low-rank PINNs containing only hundreds of model parameters” (l. 10): the parameter comparison seems unfair, since only the parameters learned during phase 2 are counted (not hypernetwork or basis vectors parameters)\n- The adaptive rank (section K) in appendix is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “low-rank PINNs containing only hundreds of model parameters” (l. 10): the parameter comparison seems unfair, since only the parameters learned during phase 2 are counted (not hypernetwork or basis vectors parameters)**\n\n**A1:** Thank you for the comment. We will make the statement more precise to be fair with baselines. Our planned edit would be as follows: \n\n“To address this issue, we propose lightweight low-rank PINNs containing only hundreds of model parameters, enabled by meta-learning a set of common basis vectors for weights and an associated hypernetwork, which allow efficient solution approximations for varying PDE input parameters.”\n\n\\\n**Q2: The adaptive rank (section K) in appendix is missing**\n\n**A2:** Thank you for pointing this out. The content of Section K was presented not in an ideal place, causing the confusion. We reorganized the sections in Appendix for better legibility.\n\n\n\\\n**Q3: \"For higher values (20), it appears that a higher rank is required to achieve a certain level of accuracy, presumably due to numerical reasons.” (l.503): what do you mean by numerical reasons?**\n\n**A3:** We wrote \"numerical reasons\" to mean that, while there is a theoretically optimal approximation that guarantees equivalent error rates for different parameter values, there is no known learning procedure that finds this optimal approximation numerically at equivalent rates of convergence. That is to say, the learning problem we consider here is known to have more challenging and non-convex loss landscapes for optimization for higher parameter values [18]. Therefore, we speculated that representing these more complex phenomena would require a higher rank, and we observed such behavior in experiments, as shown in Figure 4 of the main paper.\n\nIn the model reduction or numerical analysis literature, here a heuristic argument based on numerical experiments that suggest a trade-off between the rank and the ease of numerical approximation [Welper, 2017] unless some nonlinear transformation is applied [Rim et al, 2018].\n\nWe plan to edit the text to make this more clear, as follows: \n\n\"For higher values (20), it appears that a higher rank is required to achieve a certain level of accuracy, presumably due to the difficulty of the numerical optimization problem.\"\n\n[18] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in NeuralInformation Processing Systems, 34:26548–26560, 2021\n\n[Welper, 2017] G. Welper, Interpolation of Functions with Parameter Dependent Jumps by Transformed Snapshots, SIAM Journal on Scientific Computing 2017 39:4, A1225-A1250 \n\n[Rim et al 2023] D. Rim, B. Peherstorfer, and K.T. Mandli, Manifold Approximations via Transported Subspaces: Model Reduction for Transport-Dominated Problems, SIAM Journal on Scientific Computing 2023 45:1, A170-A199 \n\n\\\n**Q4: How does the proposed architecture scale to higher dimensions?**\n\n**A4:** For higher dimensional PDEs, we expect the specification of our LR-PINN architecture should match with the PINN architectures that are shown to be effective in approximating solutions (for example, PINNs for 1D/2D/3D PDEs in the original PINN paper). They employ relatively small-size neural networks (e.g., an MLP with 9 layers with 20 neurons) and we expect that employing a similar architecture in a low-rank format would allow approximation of solutions with reasonable accuracies. In our main paper, we used the same model architecture to learn both the 1D Convection-Diffusion-Reaction (CDR) equations and the 2D Helmholtz equation, and successfully trained the equations in both cases. Also, we expect the size of the hypernetwork can be kept relatively small as we interpret that the hypernetwork plays a role of an interpolation function. The effectiveness of lightweight hypernetworks has been demonstrated in the experiments in the manuscript. \n\n\\\n**Q5: How does the Phase 1 computation compare to the PINN variants?**\n\n**A5:** Please see global comment #1."}, {"review_id": "NLCnOPfAT4", "reviewer": "Reviewer_eb2d", "summary": "This work aims to improve on the efficiency and performance of adapting physics-informed neural networks (PINN) when dealing in many-query settings, given that the present field is limited in computationally-demanding retraining on new PDE parameters. It specifically focuses on learning low-rank versions of PINNs that enable low-parameter models through a hypernetwork-based meta-learning framework. A two-stage learning framework is proposed, in which 1) the hypernetwork and a low-rank PINN (LR-PINN) is trained across varying PDE parameters to learn a general set of basis vectors and 2) the basis vectors of the LR-PINN are fixed and the hypernetwork is used to generate an initial diagonal set on the novel PDE parameter configuration which are then trained on the new collocation points. In a set of experiments and ablations on challenging 1-D and 2-D PDEs that are known to have common \"failure modes\" for PINNs, the Hyper-LR-PINN demonstrated significant computation and performance improvements over both traditional PINN and meta-learning PINN baselines.", "questions": "** **", "limitations": "Societal impact and limitations of the model/experimentation is described appropriately, with further information on future work provided in the supplementary material.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The blend of hypernetworks and meta-learning techniques in the PINN space in order to build efficient models that can work or adapt across a variety of PDE parameters is of increasing notice and importance. This work is positioned well against similar recent works and provides an intuitive and thought-out approach for handling all of these components together, especially the smart use of hypernetworks in outputting a smaller set of important parameters as a component of the low-rank decomposition.\n\n- The writing is clean and clear and the sections are structured well.\n- The algorithm is explained clearly and it is easy to follow the all the moving components. \n- The research questions problems are clearly laid out and have corresponding answers in the model design.\n- The experiments are robust, with ablations that support each of the research question, and a comprehensive suite of baselines are considered.\n- The appendix is similarly robust, providing both visualizations and analogies to relevant background techniques that enhances understanding. The inclusion of the preliminary experiments that motivated the techniques proposed is additionally welcome.\n- The code is clean and reproducible, given provided environment files and a detailed README with instruction. All data generators are provided as well.", "weaknesses": "- The lack of explanation on the meta-learning inclusion to the framework in the main paper is a bit confusing if a reader isn't coming from a meta-PINN perspective as there are no definitions given in the standard form of meta-train and meta-test environments or details on the context/query sets. It is elaborated on more in the Appendix and it can be pieced together in the work but perhaps a more clear reference to that section in the main work or elaboration on how meta-learning is formulated in this domain may provide more general clarity. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The lack of explanation on the meta-learning inclusion to the framework in the main paper is a bit confusing if a reader isn't coming from a meta-PINN perspective as there are no definitions given in the standard form of meta-train and meta-test environments or details on the context/query sets. It is elaborated on more in the Appendix and it can be pieced together in the work but perhaps a more clear reference to that section in the main work or elaboration on how meta-learning is formulated in this domain may provide more general clarity.**\n\n**A1:** We appreciate the suggestion. We agree that readers who are not from a meta-PINN perspective may have confusion. Considering the page limitation, we think it would be better to put a clear reference to the sections in the main text. We will make this change in the manuscript at the camera-ready version. "}, {"review_id": "xMIirgJcZR", "reviewer": "Reviewer_uxdN", "summary": "In this paper, a meta-learning method is proposed for learning parameter-dependent dynamical systems. In particular, similarly to the reduced order method, the proposed method uses a low-rank neural network to construct a model, and the parameter-dependent coefficients are learned separately by a neural network. By combining these two, it is possible to retrieve the important modes, depending on the parameters of the system. This makes it possible to build models that perform well despite the small model size.", "questions": "Why does learning singular vectors not improve performance? I have an impression that this situation is similar to the reservoir computing, in which multiple random modes are first constructed and only a part of the parameters are adjusted to form a model. Should this method be understood as such a method rather than as a reduced order method?", "limitations": "No potential negative societal impact is expected.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As far as I know, modeling differential equations by low-rank neural networks is certainly new. Based on the idea of the reduced order method, I suppose that this is a natural and reliable approach. The construction of a small model is also desirable for simulation and other applications. In addition, the paper is clearly written.", "weaknesses": "Although it may not be a weakness, it is difficult to understand theoretically that learning singular vectors does not improve performance when the matrix is decomposed into lower ranks. Based on the idea of model order reduction, the singular vectors should correspond to the important modes for expressing the solution and should be learned according to the data. In this regard, there appears to be a discrepancy between the idea and the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Although it may not be a weakness, it is difficult to understand theoretically that learning singular vectors does not improve performance when the matrix is decomposed into lower ranks. Based on the idea of model order reduction, the singular vectors should correspond to the important modes for expressing the solution and should be learned according to the data. In this regard, there appears to be a discrepancy between the idea and the results.**\n\n**Why does learning singular vectors not improve performance? I have an impression that this situation is similar to the reservoir computing, in which multiple random modes are first constructed and only a part of the parameters are adjusted to form a model. Should this method be understood as such a method rather than as a reduced order method?**\n\n\n**A1:** Thank you for the comment. Indeed, this is a valid point; making the basis vector learnable in phase 2 would increase expressivity. We believe, however, the difficulty comes from the training algorithm, where the similar phenomena were observed in matrix decomposition methods. Consider finding a low-rank decomposition of matrix such that min $|| X - UDV^T ||_A$ with a norm induced from an inner product. In such a problem, updating $U,D,V$ all together in an interactive solver typically introduces more complexity. To avoid such difficulty, special solvers have been developed such as direct linear algebraic decompositions [Golub and Van Loan, 1996] which does not use a gradient-based optimization method, or notably, alternating minimization for matrix completion [Jain, et al, 2013].  In the context of low-rank approximation of solutions of PDEs, similar alternating approaches (e.g., alternating least-squares or alternating energy minimization) have been shown to be more effective [Doostan and Iaccarino, 2009], [Dolgov and Savostyanov, 2014].  \n\nDeveloping such a specialized solver in the setting of low-rank MLP can be considered as out of scope. We will mention this aspect in the limitation section where we discuss potential future directions.\n\nWe agree that there is a very high-level analogy that can be made to reservoir computing (RC), since in the second phase of our two-phase training procedure only part of the parameters are trained. However, this is a feature shared with all models making use of encoding-decoding schemes. Moreover, an important distinguishing feature in RC is the recurrent neural network, whereas our neural network architecture is feedforward. The fact that we are specifically solving parametrized partial differential equations also makes our model more specialized. We believe these points makes our model much more similar to a reduced order model, where a dimension-reduced implicit representation is learned in the first phase. In the second phase, a family of functions is quickly approximated by training a few parameters.\n\n\n[Golub and Van Loan, 1996]  Golub, Gene H. and Van Loan, Charles F., Matrix Computations, The Johns Hopkins University Press, 1996.\n\n[Jain, et al, 2013] Jain, P., Netrapalli, P., Sanghavi, S.: Low-rank matrix completion using alternating minimization. In: Proceedings of the forty-fifth annual ACM Symposium on Theory of Computing, pp. 665–674. ACM (2013)\n\n[Doostan and Iaccarino, 2009] Doostan, A., Iaccarino, G.: A least-squares approximation of partial differential equations with high-dimensional random inputs. J. Comput. Phys. 228(12), 4332–4345 (2009)\n\n[Dolgov and Savostyanov, 2014] Dolgov, S.V., Savostyanov, D.V.: Alternating minimal energy methods for linear systems in higher dimensions. SIAM J. Sci. Comput. 36(5), A2248–A2271 (2014)\n\n"}, {"review_id": "webjXYTp7e", "reviewer": "Reviewer_wUrs", "summary": "The paper presents a novel low-rank PINN architecture to solve parametrized partial differential equations (PDEs). Based on a hyper network framework they compute solutions to PDEs in a rank-adaptive way. Their approach is able to overcome for certain test systems known failure modes of PINNs. The method is extensively evaluated and benchmarked on various tasks for PDEs in 1D and 2D. ", "questions": "1.\tWhy is your model more robust against the known “failure modes”? In Table 2 it seems a Naïve LR-PINN is not able to reach the same accuracy, but considering a full rank version, it should be as expressive as the Hyper-LR-PINN approach. Could you elaborate? (related to the first bullet point in the weakness section)\n2.\tIs it clear why in the general case of CDR equations (Table 4) the improvements diminish compared to the benchmark methods?\n3.\tIs the PINN-P approach also pre-trained on multiple parametrizations in appendix N? Judging from Table 5 it seems that PINN-P is randomly initialized, but wouldn’t it be possible to pre-train it? This could potentially also improve results on PINN-P.  \n4.\tOverall, I had problems understanding how the datasets are decomposed, especially when comparing Hyper-LR-PINN with a vanilla PINN or variations (PINN-P). As far as I understood Hyper-LR-PINN is pre-trained on various parametrizations including the target parametrization and then in Phase 2 finetuned on the target. Does this mean Hyper-LR-PINN sees more datapoints on the target then the vanilla PINN?\n", "limitations": "The limitations are addressed in the main text and also the supplementary information.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Disclaimer: I am not up to date with the most recent architectural improvements regarding PINNs and state-of-the-art accuracy. \n\nTo the best of my knowledge, the architectural improvements regarding the low-rank approximation are novel and the idea to predict the singular values with a hyper network is an interesting approach to learn a neural network to represent multiple parametrizations of PDEs:\n\n1.\tThe method section is well written and easy to follow. All details are sufficiently explained, and the paper is well organized. \n2.\tThe authors carried out extensive experiments to support their architectural changes and compare it against different methods.\n3.\tThe results seem promising. Especially the results regarding the failure modes are impressive with an order of magnitude better accuracy (rel. and abs.) in Table 2 with significantly fewer parameters.  \n", "weaknesses": "The results are impressive, as mentioned above, but to me it is not clear why the hyper network is so crucial to overcome failure cases: \n\n1.\tIn my opinion the paper would benefit from a more prominent motivation, why the proposed approach is more robust against failure modes. Is it connect to the motivation in appendix D.2 and the remark in appendix B? \n2.\tIn this regard an ablation on the architectural changes would also help to better understand the improvements proposed. To my understanding the closest PINN architectures is the HyperPINN method. Would it make sense to compare it against the newly proposed ansatz for example in Table 2, as another method w/o pre-training?\n3.\tIn section 4.2.2 (“computational cost”) a fair comparison would be the meta-learning strategies, especially since some of the parametrizations of $\\beta$ are in the training set (for Phase 1). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: In my opinion the paper would benefit from a more prominent motivation, why the proposed approach is more robust against failure modes. Is it connect to the motivation in appendix D.2 and the remark in appendix B?**\n\n**A1:** Your understanding of our motivation is correct; we were motivated by the observations in appendix D.2 and the remark in appendix B, which leads to development of the proposed approach. Although we empirically presented that the proposed algorithm is effective in solving the “failure modes”, we lack theoretical explanation on the phenomena we are observing.\nWe conjecture that learning the set of basis vectors (close to orthogonal) that works well for a range of PDE parameter values seems to be the key aspect that enables the model to avoid spectral bias and, thus, facilitates approximating highly oscillatory (cf. convection equations) and very stiff (cf. reaction equations).\n\n\\\n**Q2: In this regard an ablation on the architectural changes would also help to better understand the improvements proposed. To my understanding the closest PINN architectures is the HyperPINN method. Would it make sense to compare it against the newly proposed ansatz for example in Table 2, as another method w/o pre-training?**\n\n**A2:** We indeed have a result of applying HyperPINN on the “failure mode’’ of the convection equations, shown in Appendix S. This result suggests that the improved performance of Hyper-LR-PINN is more than just using hypernetwork. As elaborated in the response to the previous comment, we believe the training algorithm accounts more for the improved performance.  \n\n\\\n**Q3: In section 4.2.2 a fair comparison would be the meta-learning strategies, especially since some of the parametrizations of $\\beta$ are in the training set (for Phase 1).**\n\n**A3:** Please see global comment #1.\n\n\\\n**Q4: Why is your model more robust against the known “failure modes”? In Table 2 it seems a Naïve LR-PINN is not able to reach the same accuracy, but considering a full rank version, it should be as expressive as the Hyper-LR-PINN approach. Could you elaborate?**\n\n**A4:** For Naïve-LR-PINN, the orthogonal basis is obtained from the randomly initialized weights of the layers. In contrast, for Hyper-LR-PINN, the basis is learned from the weights of the layers that represent the parameterized PDEs. Consequently, Hyper-LR-PINN can obtain a basis that is well-suited for representing the target parameterized PDEs. This hypernetwork based meta-learning methodology provides a favorable starting point to the model before entering the Phase 2 (See appendix R). The experiments in this paper empirically demonstrate that the obtained basis can more easily generalize to PDE coefficients that were not trained during Phase 1, as compared to Naïve-LR-PINN.\n\n\\\n**Q5: Is it clear why in the general case of CDR equations (Table 4) the improvements diminish compared to the benchmark methods?**\n\n**A5:** We believe that equations containing diffusion phenomena smooth out the solution snapshots and, thus, make the all considered methods somewhat comparable to each other. In such situations, PINNs in full-rank have a tendency to produce slightly better accuracy as they have higher expressivity. As shown in the additional comparisons (cf. Figure 1 of attached PDF), we also expect that the other competing baselines (MAML, Reptile) would have trouble finding good initializations for PINNs when the ranges of PDE coefficients become larger.\n\n\\\n**Q6: Is the PINN-P approach also pre-trained on multiple parametrizations in appendix N? Judging from Table 5 it seems that PINN-P is randomly initialized, but wouldn’t it be possible to pre-train it? This could potentially also improve results on PINN-P.**\n\n**A6:** Please see global comment #2.\n\n\\\n**Q7: Overall, I had problems understanding how the datasets are decomposed, especially when comparing Hyper-LR-PINN with a vanilla PINN or variations (PINN-P). As far as I understood Hyper-LR-PINN is pre-trained on various parametrizations including the target parametrization and then in Phase 2 finetuned on the target. Does this mean Hyper-LR-PINN sees more datapoints on the target then the vanilla PINN?**\n\n**A7:** Thanks for the comments. To address this concern, we’d like to emphasize that this is why we included baselines such as MAML and Reptile and put all those considered methods in the same experimental environments. Those are the PINN variants for solving parameterized PDEs and we can use the same setups for training/test data. All MAML,  Reptile, and the proposed Hyper-LR-PINN see the same number of coordinates (x,t) at the same number of PDE parameters ($\\mu$) during the training. \n\nFor PINN-P, thanks to the reviewer’s suggestion, we were able to perform the experiments on the same training and data setup, and reported the result in Tables 1 and 2 of the attached PDF.  With those additional experiments, we will add this description on the dataset in the manuscript to address the concern. "}]

Author rebuttal:
**Global comment #1: [Additional results of computational cost]**

Assuming that we solve PDEs for the same number of PDE parameters without our method and other baselines, the computation needed for each forward pass in Phase 1 is slightly higher than that of training all individual vanilla PINNs because the proposed method involves training the hypernetwork. At the cost of marginally increased computation, the proposed method exhibits many benefits; most prominently (1) for test PDE parameters, an error level of about 5% can be achieved within few epochs (See Figure 1 of the attached PDF) while the traditional method requires several thousand epochs, and (2) we resolve the difficulties that PINNs have in the failure modes.

Regarding the benefit (1) mentioned above, we conducted additional analyses on computational cost in a many-query scenario  for four models, PINN, MAML, Reptile, and Hyper-LR-PINN, for fair comparison. In the experiments, we trained the models for convection equations on various $\beta$ ranges, i.e., [1, 5], [1, 10], and [1, 20], and measured the number of epochs required to reach an L2 error less than 0.05. We trained three meta-learning models, MAML, Reptile, and Hyper-LR-PINN, for the convection equations with $\beta$ values with an interval of 1 in Phase 1, whereas testing the models with an interval 0.5 in Phase 2, i.e., the trained and tested $\beta$ settings do not overlap. The experimental results are summarized in Figure 1 in the attached PDF.

According to Figure 1, when learning convection equations with $\beta \in$ [1, 20] and $\beta \in$ [1, 10], the performance of two meta-learning baselines is observed to be inferior to PINN models without pre-training. Even for some beta values, the minimum threshold, i.e., an error level of about 5%, was not reached within 10,000 epochs so that they were not plotted. However, with $\beta \in$ [1, 5], the meta-learning baselines demonstrate superior performance compared to PINN in all beta values. On top of that, Hyper-LR-PINN stands out as the most effective approach, achieving the target with the least computational cost across all ranges. These experimental results indicate that existing meta-learning methodologies seem to have difficulties in finding a good “global/common” initialization for PINNs as the range of the PDE coefficients becomes larger. On the other hand, Hyper-LR-PINNs do not suffer from the same issue and are shown to be very effective over a wider range of PDE coefficients compared to existing meta-learning methods. We will include additional experimental results and discussions in the paper.

\
**Global comment #2: [Additional ablation studies on PINN-P]**

We have verified that PINN-P is amenable to pre-training since it models the parameterized solution. Following reviewers’ suggestion, we integrated an additional pre-training phase into PINN-P and conducted new experiments on convection equations with $\beta$ = {30, 35, 40}, which are PINN’s failure modes (cf. Tables 2 and 7). The fine-tuned results of PINN-P are summarized in Tables 1 and 2 from the attached PDF.

When training PINN-P, we followed the same experimental setups of experiments in Table 2. To be specific, in phase 1, we trained PINN-P using convection equations with $\beta \in$ [30, 40], and then fine-tuned the models for six specific convection equations where $\beta$ = {30, 35, 40} and $\beta$ = {41, 42, 43} (within and outside the learning range of phase 1, respectively) in phase 2, employing the pre-trained model from the phase 1. As shown in Tables 1 and 2 of the attached PDF, PINN-P with pre-training phase shows overall improvements compared to previous PINN-P experiments in Table 7. However, there still exists a substantial performance gap between PINN-P and Hyper-LR-PINN (Full rank, Adaptive rank). Both PINN-P and Hyper-LR-PINN demonstrated that the pre-training process can be beneficial in improving model performance. We will conduct further discussions on the pre-trained PINN-P model.



Meta-review: In this paper, a meta-learning framework based on the Hypernetwork is employed to propose an approach for solving PDEs using a low-rank parameterized Physics-Informed Neural Network (LR-PINN). Through a series of experiments involving 1D and 2D PDEs, significant improvements in computation and performance were demonstrated over both conventional PINNs and meta-learning PINNs as baselines. Modeling differential equations with a low-rank neural network is indeed novel. This achievement holds a strong position in comparison to recent similar research. It can be regarded as a crucial contribution that enhances both the efficiency and performance of PINNs.
Decision: Accept (spotlight)