Title: Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Scientific machine learning

Physics-informed neural networks

Meta learning

Hypernetworks
Abstract: In various engineering and applied science applications, repetitive numerical simulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest a path that potentially opens up a possibility for physics-informed neural networks (PINNs), emerging deep-learning-based solvers, to be considered as one such solver. Although PINNs have pioneered a proper integration of deep-learning and scientific computing, they require repetitive time-consuming training of neural networks, which is not suitable for many-query scenarios. To address this issue, we propose a lightweight low-rank PINNs containing only hundreds of model parameters and an associated hypernetwork-based meta-learning algorithm, which allows efficient approximation of solutions of PDEs for varying ranges of PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as "failure modes" of PINNs.

Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks

Abstract

In various engineering and applied science applications, repetitive numerical simulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest a path that potentially opens up a possibility for physics-informed neural networks (PINNs), emerging deep-learning-based solvers, to be considered as one such solver. Although PINNs have pioneered a proper integration of deep-learning and scientific computing, they require repetitive time-consuming training of neural networks, which is not suitable for many-query scenarios. To address this issue, we propose lightweight low-rank PINNs containing only hundreds of model parameters and an associated hypernetwork-based meta-learning algorithm, which allow efficient solution approximations for varying PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as 'failure modes' of PINNs.

1 Introduction

Physics-informed neural networks (PINNs) [1] are a particular class of coordinate-based multi-layer perceptrons (MLPs), also known as implicit neural representations (INRs), to numerically approximate solutions of partial differential equations (PDEs). That is, PINNs are taking spatiotemporal coordinates ( x, t ) as an input and predict PDE solutions evaluated at the coordinates u Θ ( x, t ) and are trained by minimizing (implicit) PDE residual loss and data matching loss at initial and boundary conditions. PINNs have been successfully applied to many different important applications in computational science and engineering domain: computational fluid dynamics [2, 3], cardiac electrophysiology simulation [4], material science [5], and photonics [6], to name a few.

PINNs are, however, sharing the same weakness with coordinate-based MLPs (or INRs), which hinders the application of PINNs/INRs to more diverse applications; for a new data instance (e.g., a new PDE for PINNs or a new image for INRs), training a new neural network (typically from scratch) is required. Thus, using PINNs to solve PDEs (particularly, in parameterized PDE settings) is usually computationally demanding, and this burden precludes the application of PINNs to important scenarios that involve many queries in nature as these scenarios require the parameterized PDE models to be simulated thousands of times (e.g., design optimization, uncertainty propagation), i.e., requiring PDE solutions u ( x, t ; µ ) at many PDE parameter settings { µ ( i ) } N µ i =1 with very large N µ .

To mitigate the above described issue, we propose i) a low-rank structured neural network architecture for PINNs, denoted as low-rank PINNs (LR-PINNs), ii) an efficient rank-revealing training algorithm, which adaptively adjust ranks of LR-PINNs for varying PDE inputs, and iii) a two-phase procedure (offline training/online testing) for handling many-query scenarios. This study is inspired by the observations from the studies of numerical PDE solvers [7-11] stating that numerical solutions of parametric PDEs can be often approximated in a low-rank matrix or tensor format with reduced computational/memory requirements. In particular, the proposed approach adopts the computational formalism used in reduced-order modeling (ROM) [12, 13], one of the most dominant approaches in solving parameteric PDEs, which we will further elaborate in Section 3.

In essence, LR-PINNs represent the weight of some internal layers as a low-rank matrix format. Specifically, we employ a singular-value-decomposition (SVD)-like matrix decomposition, i.e., a linear combination of rank-1 matrices: the weight of the l -th layer is W l = ∑ r i =1 s l i u l i v l i T with the rank r = min( n l , n l +1 ) , where W l ∈ R n l +1 × n l , u l i ∈ R n l +1 , v l i ∈ R n l , and s l i ∈ R . The ranks of the internal layers, however, typically are not known a priori. To address this issue, we devise a novel hypernetwork-based neural network architecture, where the rank-structure depending on the PDE parameters µ is learned via training. In short, the proposed architecture consists of i) a lightweight hypernetwork module and ii) a low-rank solution network module; the hypernetwork takes in the PDE parameters and produces the coefficients of the rank-1 series expansion (i.e., s ( µ ) = f hyper ( µ ) ). The low-rank solution network module i) takes in the spatiotemporal coordinates ( x, t ) , ii) takes the forward pass through the linear layers with the weights W l ( µ ) = ∑ r i =1 s l i ( µ ) u l i v l i T , of which s i ( µ ) comes from the hypernetwork, and iii) produces the prediction u Θ ( x, t ; µ ) . Then, the training is performed via minimizing the PINN loss, i.e., a part of the PINN loss is the PDE residual loss, ‖ R ( u Θ ( x, t ; µ ); µ ) ‖ 2 , where R ( · , · ; µ ) denotes the parameterized PDE residual operator.

We show the efficacy and the efficiency of our proposed method for solving some of the most fundamental parameterized PDEs, called convection-diffusion-reaction equations , and Helmholtz equations . Our contributions include:

We employ a low-rank neural network for PINNs after identifying three research challenges to address.

We develop a hypernetwork-based framework for solving parameterized PDEs, which computes solutions in a rank-adaptive way for varying PDE parameters.

We demonstrate that the proposed method resolves the ' failure modes ' of PINNs.

We also demonstrate that our method outperforms baselines in terms of accuracy and speed.

2 Naïve low-rank PINNs

Let us being by formally defining LR-PINNs and attempt to answer relevant research questions. LR-PINNs are a class of PINNs that has hidden fully-connected layers (FC) represented as a low-rank weight matrix. We denote this intermediate layer as LR-FC: the l -th hidden layer is defined such that

where U l r ∈ R n l +1 × r and V l r ∈ R n l × r denote full column-rank matrices (i.e., rank r /lessmuch n l , n l +1 ) containing a set of orthogonal basis vectors, and Σ l r ∈ R r × r is a diagonal matrix, Σ l r = diag ( s l r ) with s l r ∈ R r .

Memory efficiency: LR-PINNs with a rank of r and L hidden layers require O ((2 n l + 1) rL ) memory as opposed to O ( n 2 l L ) required by regular PINNs.

Computational efficiency: The forward/backward pass of LR-PINNs can be computed efficiently by utilizing a factored representation of the weights. To simplify the presentation, we describe only the forward pass computation; the forward pass is equivalent to perform three small matrix-vector products (MVPs) in sequence as indicated by the parentheses in Eq. (1).

Challenges: Representing the weights of hidden layers itself is straightforward and indeed has been studied actively in many different fields of deep learning, e.g., NLP [14, 15]. However, those approaches typically assume that there exist pre-trained models and approximate the model weights by running the truncated SVD algorithm. Our approach is different from these approaches in that we

-

-

-

...

...

attempt to reveal the ranks of internal layers as the training proceeds, which brings unique challenges. These challenges can be summarized with some research questions, which include: C1 ) 'should we make all parameters learnable (i.e., ( U l , V l , Σ l ) )?', C2 ) 'how can we determine the ranks of each layer separately, and also adaptively for varying µ ?', and C3 ) 'can we utilize a low-rank structure to avoid expensive and repetitive training of PINNs for every single new µ instances?'. In the following, we address these questions by proposing a novel neural network architecture.

3 Hyper-LR-PINNs: hypernetwork-based meta-learning low-rank PINNs

We propose a novel neural network architecture based on a hypernetwork and an associated training algorithm, which address the predescribed challenges. To distinguish them, we hereinafter call LRPINN with (resp. w/o) our proposed hypernetwork as 'Hyper-LR-PINN' (resp. 'Naïve-LR-PINN').

Design goals: Here we attempt to resolve all challenges by setting up several design goals that are inspired from our domain knowledge in the field (and also from some preliminary results that can be found in Appendix D):

build a single set of basis vectors U l and V l , preferably as orthogonal as possible, that perform well over a range of PDE parameters,

build LR-PINNs with an adaptive and layer-wise rank structure that depends on PDE parameters (e.g., a higher rank for a higher convective PDE) and,

make only the diagonal elements , denoted s l r in Eq. (1), learnable to achieve high efficiency once a proper set of basis vectors and the rank structure are identified.

Our design largely follows the principles of ROMs, where the expensive computation is offloaded to an offline phase to build a cheap surrogate model that can perform an efficient computation in an online phase for a test set. To make an analogy, consider parameterized dynamical systems (which may arise in semi-discretization of time-dependent PDEs): d u ( t ; µ ) d t = f ( u ( t ; µ ); µ ) , where u ∈ R N . In the offline phase, the method seeks a low-dimensional linear trial basis where the reduced representation of the solution lie on, which is achieved by performing high-fidelity simulations on a set of PDE parameter instances { µ ( i ) } n train i =1 and constructing a trial linear subspace Ψ := range ( Ψ p ) with Ψ p = [ ψ 1 , · · · , ψ p ] ∈ R N × p from the solution snapshots collected from the high-fidelity simulations. In the online phase, the solutions at a set of test PDE parameter instances { µ ( i ) } n test i =1 are approximated as u ( t, µ ) ≈ Ψ p c ( t, µ ) with c ∈ R p , and a low-dimensional surrogate problem is derived as d c ( t ; µ ) d t = Ψ T p f ( Ψ p c ( t ; µ ); µ ) = ˆ f ( c ( t ; µ ); µ ) ∈ R p , which can be rapidly solved, while not losing too much accuracy. See Appendix C for an illustrative explanation and details of ROMs.

Taking a cue from the ROM principles, we design our model to operate on a common set of basis vectors { U l r } and { V l r } , which are obtained during the offline phase (analogous to Ψ p in ROMs) and update only the diagonal elements { s l r } during the online phase (analogous to c in ROMs). Now, we elaborate our network design and the associated two-phase algorithm. For the connection to the ROM, which explains details on the context/query sets, can be found in Appendix C.

3.1 Hypernetwork-based neural network architecture

The proposed framework has two computational paths: a path for the hypernetwork and a path for LR-PINN (Figure 1). The hypernetwork path reads the PDE parameter µ and outputs the diagonal elements { s l } L l =1 of LR-FCs. The LR-PINN path reads ( x, t ) and the output of the hypernetwork, and outputs the approximated solution u Θ at ( x, t ; µ ) , which can be written as follows:

where f hyper ( µ ) denotes the hypernetwork such that { s l ( µ ) } L l =1 = f hyper ( µ ) . We denote s as a function of µ to make it explicit that it is dependent on µ . The internals of LR-PINN can be described as with h 0 = [ x, t ] T

where Σ l ( µ ) = diag ( s l ( µ )) . The hypernetwork can be described as the following initial embedding layer, where e 0 = µ , followed by an output layer:

where ReLU is employed to automatically truncate the negative values so that the adaptive rank structure for varying PDE parameters can be revealed (i.e., the number of non-zeros (NNZs) 2 in s l ( µ ) varies depending on µ ).

3.2 Two-phase training algorithm

Along with the framework, we present the proposed two-phase training algorithm. Phase 1 is for learning the common set of basis vectors and the hypernetwork and Phase 2 is for fine-tuning the network for a specific set of test PDE parameters. Table 1 shows the sets of model parameters that are being trained in each phase. (See Appendix E for the formal algorithm.)

In Phase 1, we train the hypernetwork and the LR-PINN jointly on a set of collocation points that are collected for varying PDE parameters. Through the computational procedure described in Section 3.1, the approximated solutions at the collocation points are produced u Θ (( x j , t j ); µ ( i ) ) . Then, as in regular PINNs, the PDE residual loss and the data matching loss can be computed. The small difference is that the PDE operator, F , for the residual loss is also parameterized such that F ( u Θ (( x j , t j ); µ ( i ) ); µ ( i ) ) . As we wish to obtain basis vectors that are close to orthogonal, we add the following orthogonality constraint based on the Frobenius norm to the PINN loss [16]:

where w 1 and w 2 are penalty weights. (See Appendix J.)

In Phase 2, we continue training LR-PINN for approximating the solutions of a target PDE parameter configuration. We i) fix the weights, the biases, and the set of basis vectors of LR-FC obtained from Phase 1 , ii) convert the diagonal elements to a set of learnable parameters after initializing them with the values from the hypernetwork, and iii) detach the hypernetwork. Thus, only the trainable parameters from this point are the set of diagonal elements, first and last linear layers. The hypernetwork-initialized diagonal elements serve as a good starting point in (stochastic) gradient update optimizers (i.e., require less number of epochs). Moreover, significant computational savings can be achieved in the gradient update steps as only the diagonal elements are updated (i.e., s l +1 ← s l + η ∇ s L instead of Θ l +1 ← Θ l + η ∇ Θ L ).

4 Experiments

We demonstrate that our proposed method significantly outperforms baselines on the 1-dimensional/2dimensional PDE benchmarks that are known to be very challenging for PINNs to learn [17, 18]. We report the average accuracy and refer readers to Appendix U for the std. dev. of accuracy after 3 runs.

Phase 1

{

(

U

l

, V

l

, b

l

)

}

L

l

=1

,

W

0

, W

L

+1

, b

0

, b

L

+1

(LR-PINN),

{

(

W

emb

,m

, b

emb

,m

)

}

M

m

=1

,

{

(

W

hyper

,l

, b

hyper

,l

)

}

L

l

=1

(hypernetwork),

Phase 2

{

s

l

}

L

l

=1

,

W

0

, W

L

+1

, b

0

, b

L

+1

(LR-PINN)

(a) Exact solution

(b) PINN

(c) Ours

2.0

6

1.7

1.5

4

1.2

×

1.0

0.7

2

0.5

0.2

8.0

0.2

0.4

0.6

0.8

1.0

0.0

2.0

6

1.7

1.5

4

1.2

×

1.0

0.7

2

0.5

0.2

8.0

0.2

0.4

0.6

0.8

1.0

0.0

2.0

6

1.7

1.5

4

1.2

×

1.0

0.7

2

0.5

0.2

8.0

0.2

0.4

0.6

0.8

1.0

0.0

Baselines for comparison: Along with the vanilla PINN [1], our baselines include several variants. PINN-R [19] denotes a model that adds skip-connections to PINN. PINN-S2S [18] denotes a method that uniformly segments the temporal domain and proceeds by training each segment one by one in a temporal order. PINN-P denotes a method that directly extends PINNs to take ( x, t, µ ) as input and infer u θ ( x, t, µ ) , i.e., µ is being treated as a coordinate in the parameter domain.

We also apply various meta-learning algorithms to Naïve-LR-PINN: model-agnostic meta learning (MAML) [20] and Reptile [21] - recall that Naïve-LR-PINN means that LR-PINN without our proposed hypernetwork-based meta-learning and therefore, MAML and Reptile on top of Naïve-LRPINN can be compared to Hyper-LR-PINN. In the parameterized PDE setting, we can define a task, τ ( i ) , as a specific setting of the PDE parameters, µ ( i ) . Both MAML and Reptile seek initial weights of a PINN, which can serve as a good starting point for gradient-based optimizers when a solution of a new unseen PDE parameter setting is sought. See Appendix G for details. For reproducibility, we refer readers to Appendix F, including hyperparameter configuration and software/hardware environments.

Evaluation metrics: Given the i -th PDE parameter instance µ ( i ) , the ground-truth solution evaluated at the set of test collocation points can be defined collectively as u ( i ) = [ u ( x 1 , t 1 ; µ ( i ) ) , . . . , u ( x N , t N ; µ ( i ) )] T and likewise for PINNs as u ( i ) θ . Then the absolute error and the relative error can be defined as 1 N ‖ u ( i ) -u ( i ) θ ‖ 1 and ‖ u ( i ) -u ( i ) θ ‖ 2 / ‖ u ( i ) ‖ 2 , respectively. In Appendix N, we measure the performance on more metric: max error and explained variance score.

4.1 Benchmark parameterized PDEs

1D PDEs: For our first benchmark parameterized PDEs, we consider the following parameterized convection-diffusion-reaction (CDR) equation:

where the equation describes how the state variable u evolves under convective (the second term), diffusive (the third term), and reactive (the fourth term) phenomena. 3 The triplet, µ = ( β , ν , ρ ) , defines the characteristics of the CDR equations: how strong convective/diffusive/reactive the equation is, respectively. A particular choice of µ leads to a realization of a specific type of CDR processes.

There is a set of µ which makes training PINNs very challenging, known as 'failure modes' [18]: i) convection equations with high convective terms ( β ≥ 30 ) and ii) reaction(-diffusion) equations with high reactive terms ( ρ ≥ 5 ). We demonstrate that our method does not require specific-PDE-dedicated algorithms (e.g., [18]) while producing comparable/better accuracy in low rank.

(a) Train loss

(b) The absolute error

(c) The relative error

2D PDEs: As the second set of benchmarks, we consider the 2-dim parameterized Helmholtz equation:

where q ( x, y ; a 1 , a 2 ) denotes a specific parameterized forcing term and the solution u can be calculated analytically (See Appendix T). As observed in the failure modes of convection equations (high convective terms), certain choices of the parameters in the forcing term, i.e., a 1 , a 2 , make the training of PINNs challenging as the solutions become highly oscillatory.

4.2 Experimental results

4.2.1 Performance on the failure mode of the CDR equation

Solution accuracy: As studied in prior work, vanilla PINNs fail to approximate solutions exhibiting either highly oscillatory (due to high convection) or sharp transient (due to high reaction) behaviors. Here, we present the results of convection equations and leave the results on reaction(-diffusion) equations in Appendix N. For convection equations 4 , failures typically occur with high β , e.g., β ≥ 30 . Table 2 reports the results of all considered models trained on β ∈ [30 , 40] 5 and essentially shows that Hyper-LR-PINN (Figure 2(c)) is the only low-rank method that can resolve the failuremode and there are only marginal decreases in accuracy compared to Hyper-LR-PINN in full rank. For reaction(-diffusion) equations, we observe similar results, Hyper-LR-PINN outperforms the

∼

GLYPH<c=6,font=/AAAABI+DejaVuSans>GLYPH<c=4,font=/AAAABI+DejaVuSans>

GLYPH<c=6,font=/AAAABI+DejaVuSans>GLYPH<c=5,font=/AAAABI+DejaVuSans>

GLYPH<c=6,font=/AAAABI+DejaVuSans>GLYPH<c=7,font=/AAAABI+DejaVuSans>

GLYPH<c=6,font=/AAAABI+DejaVuSans>GLYPH<c=8,font=/AAAABI+DejaVuSans>

GLYPH<c=6,font=/AAAABI+DejaVuSans>GLYPH<c=10,font=/AAAABI+DejaVuSans>

GLYPH<c=7,font=/AAAABI+DejaVuSans>GLYPH<c=4,font=/AAAABI+DejaVuSans>

GLYPH<c=4,font=/AAAABJ+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAABI+DejaVuSans>GLYPH<c=9,font=/AAAABI+DejaVuSans>

GLYPH<c=5,font=/AAAABI+DejaVuSans>GLYPH<c=10,font=/AAAABI+DejaVuSans>

GLYPH<c=5,font=/AAAABI+DejaVuSans>GLYPH<c=11,font=/AAAABI+DejaVuSans>

GLYPH<c=12,font=/AAAABI+DejaVuSans> GLYPH<c=14,font=/AAAABI+DejaVuSans>GLYPH<c=13,font=/AAAABI+DejaVuSans>GLYPH<c=19,font=/AAAABI+DejaVuSans>GLYPH<c=21,font=/AAAABI+DejaVuSans>GLYPH<c=16,font=/AAAABI+DejaVuSans>GLYPH<c=22,font=/AAAABI+DejaVuSans>GLYPH<c=15,font=/AAAABI+DejaVuSans>GLYPH<c=3,font=/AAAABI+DejaVuSans>GLYPH<c=20,font=/AAAABI+DejaVuSans>GLYPH<c=13,font=/AAAABI+DejaVuSans>GLYPH<c=18,font=/AAAABI+DejaVuSans>GLYPH<c=17,font=/AAAABI+DejaVuSans>

(a)

β

[30,40]

GLYPH<c=5,font=/AAAABK+DejaVuSans>

GLYPH<c=8,font=/AAAABK+DejaVuSans>

GLYPH<c=10,font=/AAAABK+DejaVuSans>

GLYPH<c=5,font=/AAAABK+DejaVuSans>GLYPH<c=4,font=/AAAABK+DejaVuSans>

GLYPH<c=5,font=/AAAABK+DejaVuSans>GLYPH<c=7,font=/AAAABK+DejaVuSans>

GLYPH<c=5,font=/AAAABK+DejaVuSans>GLYPH<c=9,font=/AAAABK+DejaVuSans>

GLYPH<c=5,font=/AAAABK+DejaVuSans>GLYPH<c=12,font=/AAAABK+DejaVuSans>

GLYPH<c=4,font=/AAAABL+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAABK+DejaVuSans>GLYPH<c=4,font=/AAAABK+DejaVuSans>

GLYPH<c=6,font=/AAAABK+DejaVuSans>GLYPH<c=6,font=/AAAABK+DejaVuSans>

GLYPH<c=6,font=/AAAABK+DejaVuSans>GLYPH<c=8,font=/AAAABK+DejaVuSans>

GLYPH<c=6,font=/AAAABK+DejaVuSans>GLYPH<c=9,font=/AAAABK+DejaVuSans>

GLYPH<c=6,font=/AAAABK+DejaVuSans>GLYPH<c=11,font=/AAAABK+DejaVuSans>

GLYPH<c=7,font=/AAAABK+DejaVuSans>GLYPH<c=4,font=/AAAABK+DejaVuSans>

GLYPH<c=7,font=/AAAABK+DejaVuSans>GLYPH<c=6,font=/AAAABK+DejaVuSans>

GLYPH<c=7,font=/AAAABK+DejaVuSans>GLYPH<c=8,font=/AAAABK+DejaVuSans>

GLYPH<c=13,font=/AAAABK+DejaVuSans> GLYPH<c=15,font=/AAAABK+DejaVuSans>GLYPH<c=14,font=/AAAABK+DejaVuSans>GLYPH<c=20,font=/AAAABK+DejaVuSans>GLYPH<c=22,font=/AAAABK+DejaVuSans>GLYPH<c=17,font=/AAAABK+DejaVuSans>GLYPH<c=23,font=/AAAABK+DejaVuSans>GLYPH<c=16,font=/AAAABK+DejaVuSans>GLYPH<c=3,font=/AAAABK+DejaVuSans>GLYPH<c=21,font=/AAAABK+DejaVuSans>GLYPH<c=14,font=/AAAABK+DejaVuSans>GLYPH<c=19,font=/AAAABK+DejaVuSans>GLYPH<c=18,font=/AAAABK+DejaVuSans>

(b)

β

[1,20]

GLYPH<c=7,font=/AAAABM+DejaVuSans>

GLYPH<c=14,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=11,font=/AAAABM+DejaVuSans>

GLYPH<c=8,font=/AAAABM+DejaVuSans>GLYPH<c=8,font=/AAAABM+DejaVuSans>

GLYPH<c=8,font=/AAAABM+DejaVuSans>GLYPH<c=15,font=/AAAABM+DejaVuSans>

GLYPH<c=9,font=/AAAABM+DejaVuSans>GLYPH<c=12,font=/AAAABM+DejaVuSans>

GLYPH<c=10,font=/AAAABM+DejaVuSans>GLYPH<c=9,font=/AAAABM+DejaVuSans>

GLYPH<c=11,font=/AAAABM+DejaVuSans>GLYPH<c=6,font=/AAAABM+DejaVuSans>

GLYPH<c=4,font=/AAAABN+DejaVuSans-Bold>

GLYPH<c=8,font=/AAAABM+DejaVuSans>

GLYPH<c=4,font=/AAAABM+DejaVuSans>

GLYPH<c=6,font=/AAAABN+DejaVuSans-Bold>

GLYPH<c=5,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>

GLYPH<c=9,font=/AAAABM+DejaVuSans>

GLYPH<c=11,font=/AAAABM+DejaVuSans>

GLYPH<c=13,font=/AAAABM+DejaVuSans>

GLYPH<c=15,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=7,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=9,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=11,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=13,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=15,font=/AAAABM+DejaVuSans>

GLYPH<c=4,font=/AAAABO+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAABM+DejaVuSans>

GLYPH<c=8,font=/AAAABM+DejaVuSans>

GLYPH<c=10,font=/AAAABM+DejaVuSans>

GLYPH<c=12,font=/AAAABM+DejaVuSans>

GLYPH<c=14,font=/AAAABM+DejaVuSans>

GLYPH<c=7,font=/AAAABM+DejaVuSans>GLYPH<c=6,font=/AAAABM+DejaVuSans>

(c)

β

[1,20]

∈

∈

∈

best baseline by more than an order of magnitude (See Appendix N). Moreover, we report that Hyper-LR-PINN outperforms in most cases to the baselines that operate in 'full-rank': meta-learning methods (MAML, Reptile), PINN-S2S, and PINN-P (Appendix M).

Loss curves: Figure 3 depicts two curves of train loss and test errors as the meta-learning algorithms proceed (MAML, Reptile, and ours). As opposed to the optimization-based meta-learning algorithms, which tend to learn meta-initial weights that perform well 'on average' over randomly sampled training tasks, our hypernetwork-based method minimizes the loss for each individual task simultaneously. Loss curves for Phase 2 are reported in Appendix O, which essentially show that the baseline meta-learners do not provide good initializations.

Rank structure: Table 3 compares the number of trainable model parameters for Naïve-LR-PINN, PINN, and our method. In our model, each hidden layer has a different rank structure, leading to 351 trainable parameters, which is about ( × 30) smaller than that of the vanilla PINN, Cf. merely decreasing the model size of PINNs leads to poor performance (Appendix S). Figure 4(a) shows how the model adaptively learns the rank structure for varying values of β . We observe that, with the low-rank format, Hyper-LR-PINN provides more than ( × 4 ) speed up in training time compared to the vanilla PINN. More comparisons on this aspect will be presented in the general CDR settings.

Ablation on fixed or learnable basis vectors: We compare the performance of the cases, where the basis vectors { U l r , V l r } are trainable. We observe that the fixed basis vectors lead to an order of magnitude more accurate predictions as well as faster convergence (See Appendix Q for plots).

Ablation on orthogonality constraint: We compare the performances of the cases, where the proposed model is trained without the orthogonality constraint, Eq. (2). Experimental results show that the orthogonality constraint is essential in achieve good performance (See Appendix J).

Many-query settings (inter/extra-polation in the PDE parameter domain): Next, we test our algorithm on the many-query scenario, where we train the proposed framework with the 11 training PDE parameters β ∈ [30 , 40] (Phase 1), freeze the hypernetwork and the learned basis, and retrain only the diagonal elements for 150 unseen test PDE parameters (Phase 2). Figure 5 depicts the absolute and relative errors of the solutions, which are as accurate as the ones on the training PDE parameters, in particular, for β < 30 (i.e., extrapolation). Although the model produces an increasing error for β > 40 , the relative error is still around ∼ 10%, which cannot be achievable by vanilla PINNs. The right chart of Figure 5 shows the number of trainable model parameters, which is determined by the rank (i.e., the output of the hypernetwork). As PINNs fail to produce any reasonable approximation in this β regime, we make comparisons in the general CDR settings below.

(a) The absolute error

(b) The relative error

(c) Model size

4.2.2 Performance on the general case of the CDR equation

Solution accuracy: Next, we present the performance of the proposed method on solving the benchmark PDEs (not only in the failure mode but) in all available settings for β , ν , ρ . Due to space reasons, we present only a subset of our results and leave the detailed results in Appendix M. Table 4 shows that the proposed Hyper-LR-PINN mostly outperforms the baselines that operate in full rank.

Some observations on learned low-rank structures: Again, the proposed method learns rank structures that are adaptive to the PDE parameter (Figure 4(b)). The learned rank structures vary more dynamically ( r from 20 to 34) as the PDE parameter range [1,20] is larger compared to [30,40].

Figure 4(c) visualizes the magnitude of the learned diagonal elements s 2 in the second hidden layer. The horizontal axis indicates the index of the element in s 2 , which is sorted in descending order for analysis, and the vertical axis indicates the β value. The depicted result agrees with our intuitions in many aspects: i) the values decay fast (indicating there exist low-rank structures), ii) the value of each element s 2 i either increases or decreases gradually as we vary β , and iii) for higher β , higher frequency basis vectors are required to capture more oscillatory behavior exhibited in the solutions, which leads to higher ranks. We report the information for other layers and other problems in Appendix L.

Computational cost: Here, we compare the computational cost of Hyper-LR-PINN and PINN in a many-query scenario. We train HyperLR-PINN for β ∈ [1 , 20] with interval 1 (Phase 1) and perform Phase 2 on β ∈ [1 , 20] with interval 0.5 (including unseen test PDE parameters). We record the number of epochs required for PINN and Hyper-LR-PINN to reach an L2 absolute error of less than 0.05. Figure 6 reports the number of required epochs for each β , showing that the number of epochs required for Hyper-LR-PINN slowly increases while that for PINNs increases rapidly; for most cases, Hyper-LR-PINN reaches the target (i.e., 0.05) in only one epoch. We also emphasize that Hyper-LR-PINN in Phase 2 requires much less FLOPS in each epoch (See Section 3.2).

4.2.3 Performance on the 2D Helmholtz equation

Now we report experimental results on 2D Helmholtz equations for varying a 1 and a 2 . Under the same condition in Table 2, we train Hyper-LR-PINN for a ∈ [2 , 3] with interval 0.1 ( a = a 1 = a 2 ) , and compare the performance with PINN. Surprisingly, Hyper-LR-PINN approximates the solution

(a) PINN (Abs.err.=0.7403)

(b) Ours (Abs.err.=0.0285)

(c) Exact solution

1.0

1.00

0.75

0.5

0.50

0.25

0.0

0.00

-0.2

0.5

-0.5

-0.7

1.91.0

-0.5

0.0

0.5

1.0

-1.0

1.0

1.00

0.75

0.5

0.50

0.25

0.0

0.00

0.

0.5

-0.

-0.

1.091.0

-0.5

0.0

0.5

1.0

-1.

1.0

1.00

0.75

0.5

0.50

0.25

0.0

0.00

-0.2

0.5

-0.5

-0.7

1.91.0

-0.5

0.0

0.5

1.0

-1.0

with high precision in only 10 epochs in Phase 2, while PINN struggles to find accurate solutions over 2,000 epochs (Figure 7). We show more visualizations and detailed results in Appendix T.

5 Related Work

Meta-learning of PINNs and INRs: HyperPINNs [22] share some commonalities with our proposed method in that they generate model parameters of PINNs via hypernetwork [23]. However, hyperPINNs can only generate full-rank weights and do not have capabilities to handle parameterized PDEs. Another relevant work is optimization-based meta-learning algorithms; representatively MAML[20] and Reptile [21]. In the area of INRs, meta-learning methods via MAML and Reptile for obtaining initial weights for INRs have been studied in [24]. In [25, 26], meta-learning methods, which are based on MAML, have been further extended to obtain sparse representation of INRs.

Low-rank formats in neural networks: In natural language processing, the models being employed (e.g., Bert) typically have hundreds of millions of model parameters and, thus, making the computation efficient during the inference is one of the imminent issues. As a remedy, approximating layers in low-rank via truncated SVD has been studied [14, 15]. Modeling layers in low-rank in general has been studied for MLPs [27-30] and convolutional neural network architectures [31]. In the context of PINNs or INRs, there is no low-rank format has been investigated. The work that is the closest to our work is SVD-PINNs [32], which represents the hidden layers in the factored form as in Eq. (1), but always in full-rank.

PINNs and their variants: There have been numerous sequels improving PINNs [1] in many different aspects. One of the main issues is the multi-term objectives; in [19], a special gradientbased optimizer that balances multiple objective terms has been proposed, and in [33], a network architecture that enforce boundary conditions by design has been studied. Another main issue is that training PINNs often fails on certain classes of PDEs (e.g., fail to capture sharp transition in solutions over the spatial/temporal domains). This is due to the spectral bias [34] and, as a remedy, in [18], training approaches that gradually increase the level of difficulties in training PINNs have been proposed. In [35], a Lagrangian-type reformulation of PINNs is proposed for convection-dominated PDEs. There are also other types of improvements including Bayesian-version of PINNs [36] and PINNs that enforce conservation laws [37]. All these methods, however, require training from scratch when the solution of a new PDE is sought.

6 Conclusion

In this paper, we propose a low-rank formatted physics-informed neural networks (PINNs) and a hypernetwork-based meta-learning algorithm to solve parameterized partial differential equations (PDEs). Our two-phase method learns a common set of basis vectors and adaptive rank structure for varying PDE parameters in Phase 1 and approximate the solutions for unseen PDE parameters by updating only the coefficients of the learned basis vectors in Phase 2. From the extensive numerical experiments, we have demonstrated that the proposed method outperforms baselines in terms of accuracy and computational/memory efficiency and does not suffer from the failure modes for various parameterized PDEs.

Limitations: Our Hyper-LR-PINN primarily concentrates on the cases where the PDE parameters are the coefficients of PDE terms. As a future study, we plan to extend our framework to more general settings, where the PDE parameters define initial/boundary conditions, or the shape of domains. See Appendix A for more details.

Broader impacts: Hyper-LR-PINN can solve many equations even for PINN's failure modes with high precision. However, despite its small errors, one should exercise caution when applying it to real-world critical applications.

Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program ay Yonsei University, 10%), and (No.2022-0-00857, Development of AI/databased financial/economic digital twin platform, 90%).

References

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: Adeep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering , 360:112789, 2020.

XIA Yang, Suhaib Zafar, J-X Wang, and Heng Xiao. Predictive large-eddy-simulation wall modeling via physics-informed neural networks. Physical Review Fluids , 4(3):034602, 2019.

Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics , 8:42, 2020.

Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using physics-informed neural networks. Science advances , 8(7):eabk0644, 2022.

Wei Ma, Zhaocheng Liu, Zhaxylyk A Kudyshev, Alexandra Boltasseva, Wenshan Cai, and Yongmin Liu. Deep learning for the design of photonic structures. Nature Photonics , 15(2):7790, 2021.

Daniel Kressner and Christine Tobler. Low-rank tensor Krylov subspace methods for parametrized linear systems. SIAM Journal on Matrix Analysis and Applications , 32(4):12881316, 2011.

Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen , 36(1):53-78, 2013.

Kookjin Lee and Howard C Elman. A preconditioned low-rank projection method with a rank-reduction scheme for stochastic partial differential equations. SIAM Journal on Scientific Computing , 39(5):S828-S850, 2017.

Markus Bachmayr, Albert Cohen, and Wolfgang Dahmen. Parametric PDEs: Sparse or low-rank approximations? IMA Journal of Numerical Analysis , 38(4):1661-1708, 2018.

Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics , 404:108973, 2020.

Philip Holmes, John L Lumley, Gahl Berkooz, and Clarence W Rowley. Turbulence, coherent structures, dynamical systems and symmetry . Cambridge university press, 2012.

Peter Benner, Serkan Gugercin, and Karen Willcox. A survey of projection-based model reduction methods for parametric dynamical systems. SIAM review , 57(4):483-531, 2015.

Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data-aware lowrank compression for large NLP models. Advances in neural information processing systems , 34:29321-29334, 2021.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021.

Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 678-679, 2020.

Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.04544 , 2020.

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin, and Noseong Park. Dpm: A novel training method for physics-informed neural networks in extrapolation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 8146-8154, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Filipe de Avila Belbute-Peres, Yi-fan Chen, and Fei Sha. HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks. In The Symbiosis of Deep Learning and Differential Equations , 2021.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 ,

2016.

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2846-2855, 2021.

Jaeho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems , 34:11769-11780, 2021.

Jonathan Richard Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. arXiv preprint arXiv:2205.08957 , 2022.

Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 6655-6659. IEEE, 2013.

Yu Zhang, Ekapol Chuangsuwanich, and James Glass. Extracting deep neural network bottleneck features using low-rank matrix factorization. In 2014 IEEE international conference on acoustics, speech and signal processing , pages 185-189. IEEE, 2014.

Yong Zhao, Jinyu Li, and Yifan Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5005-5009. IEEE, 2016.

Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations , 2020.

Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press , 2014.

Yihang Gao, Ka Chun Cheung, and Michael K Ng. SVD-PINNs: Transfer learning of physicsinformed neural networks via singular value decomposition. arXiv preprint arXiv:2211.08760 , 2022.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055-A3081, 2021.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2019.

Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Kolmogorov n -width and Lagrangian physics-informed neural networks: A causality-conforming manifold for convectiondominated pdes. Computer Methods in Applied Mechanics and Engineering , 404:115810, 2023.

Liu Yang, Xuhui Meng, and George Em Karniadakis. B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. Journal of Computational Physics , 425:109913, 2021.

Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering , 365:113028, 2020.

Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013.

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 665-674, 2013.

Alireza Doostan and Gianluca Iaccarino. A least-squares approximation of partial differential equations with high-dimensional random inputs. Journal of computational physics , 228(12):4332-4345, 2009.

Sergey V Dolgov and Dmitry V Savostyanov. Alternating minimal energy methods for linear systems in higher dimensions. SIAM Journal on Scientific Computing , 36(5):A2248-A2271, 2014.

Kookjin Lee, Howard C Elman, Catherine E Powell, and Dongeun Lee. Enhanced alternating energy minimization methods for stochastic galerkin matrix equations. BIT Numerical Mathematics , 62(3):965-994, 2022.

K. Dingle, C. Q. Camargo, and A. A. Louis. Input-output maps are strongly biased towards simple outputs. Nature communications , 2018.

Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020.

Minyoung Huh, Hossein Mobahi, Richard Zhang, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv , 2021.

Benjamin Schrauwen, David Verstraeten, and Jan Van Campenhout. An overview of reservoir computing: theory, applications and implementations. In Proceedings of the 15th european symposium on artificial neural networks. p. 471-482 2007 , pages 471-482, 2007.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel low-rank PINN architecture to solve parametrized partial differential equations (PDEs). Based on a hyper network framework they compute solutions to PDEs in a rank-adaptive way. Their approach is able to overcome for certain test systems known failure modes of PINNs. The method is extensively evaluated and benchmarked on various tasks for PDEs in 1D and 2D.\n\nSTRENGTHS:\nDisclaimer: I am not up to date with the most recent architectural improvements regarding PINNs and state-of-the-art accuracy. \n\nTo the best of my knowledge, the architectural improvements regarding the low-rank approximation are novel and the idea to predict the singular values with a hyper network is an interesting approach to learn a neural network to represent multiple parametrizations of PDEs:\n\n1.\tThe method section is well written and easy to follow. All details are sufficiently explained, and the paper is well organized. \n2.\tThe authors carried out extensive experiments to support their architectural changes and compare it against different methods.\n3.\tThe results seem promising. Especially the results regarding the failure modes are impressive with an order of magnitude better accuracy (rel. and abs.) in Table 2 with significantly fewer parameters.\n\nWEAKNESSES:\nThe results are impressive, as mentioned above, but to me it is not clear why the hyper network is so crucial to overcome failure cases: \n\n1.\tIn my opinion the paper would benefit from a more prominent motivation, why the proposed approach is more robust against failure modes. Is it connect to the motivation in appendix D.2 and the remark in appendix B? \n2.\tIn this regard an ablation on the architectural changes would also help to better understand the improvements proposed. To my understanding the closest PINN architectures is the HyperPINN method. Would it make sense to compare it against the newly proposed ansatz for example in Table 2, as another method w/o pre-training?\n3.\tIn section 4.2.2 (“computational cost”) a fair comparison would be the meta-learning strategies, especially since some of the parametrizations of $\\beta$ are in the training set (for Phase 1).\n\nQUESTIONS:\n1.\tWhy is your model more robust against the known “failure modes”? In Table 2 it seems a Naïve LR-PINN is not able to reach the same accuracy, but considering a full rank version, it should be as expressive as the Hyper-LR-PINN approach. Could you elaborate? (related to the first bullet point in the weakness section)\n2.\tIs it clear why in the general case of CDR equations (Table 4) the improvements diminish compared to the benchmark methods?\n3.\tIs the PINN-P approach also pre-trained on multiple parametrizations in appendix N? Judging from Table 5 it seems that PINN-P is randomly initialized, but wouldn’t it be possible to pre-train it? This could potentially also improve results on PINN-P.  \n4.\tOverall, I had problems understanding how the datasets are decomposed, especially when comparing Hyper-LR-PINN with a vanilla PINN or variations (PINN-P). As far as I understood Hyper-LR-PINN is pre-trained on various parametrizations including the target parametrization and then in Phase 2 finetuned on the target. Does this mean Hyper-LR-PINN sees more datapoints on the target then the vanilla PINN?\n\nLIMITATIONS:\nThe limitations are addressed in the main text and also the supplementary information.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

comments: SUMMARY:
In this paper, a meta-learning method is proposed for learning parameter-dependent dynamical systems. In particular, similarly to the reduced order method, the proposed method uses a low-rank neural network to construct a model, and the parameter-dependent coefficients are learned separately by a neural network. By combining these two, it is possible to retrieve the important modes, depending on the parameters of the system. This makes it possible to build models that perform well despite the small model size.

STRENGTHS:
As far as I know, modeling differential equations by low-rank neural networks is certainly new. Based on the idea of the reduced order method, I suppose that this is a natural and reliable approach. The construction of a small model is also desirable for simulation and other applications. In addition, the paper is clearly written.

WEAKNESSES:
Although it may not be a weakness, it is difficult to understand theoretically that learning singular vectors does not improve performance when the matrix is decomposed into lower ranks. Based on the idea of model order reduction, the singular vectors should correspond to the important modes for expressing the solution and should be learned according to the data. In this regard, there appears to be a discrepancy between the idea and the results.

QUESTIONS:
Why does learning singular vectors not improve performance? I have an impression that this situation is similar to the reservoir computing, in which multiple random modes are first constructed and only a part of the parameters are adjusted to form a model. Should this method be understood as such a method rather than as a reduced order method?

LIMITATIONS:
No potential negative societal impact is expected.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to improve on the efficiency and performance of adapting physics-informed neural networks (PINN) when dealing in many-query settings, given that the present field is limited in computationally-demanding retraining on new PDE parameters. It specifically focuses on learning low-rank versions of PINNs that enable low-parameter models through a hypernetwork-based meta-learning framework. A two-stage learning framework is proposed, in which 1) the hypernetwork and a low-rank PINN (LR-PINN) is trained across varying PDE parameters to learn a general set of basis vectors and 2) the basis vectors of the LR-PINN are fixed and the hypernetwork is used to generate an initial diagonal set on the novel PDE parameter configuration which are then trained on the new collocation points. In a set of experiments and ablations on challenging 1-D and 2-D PDEs that are known to have common \"failure modes\" for PINNs, the Hyper-LR-PINN demonstrated significant computation and performance improvements over both traditional PINN and meta-learning PINN baselines.\n\nSTRENGTHS:\nThe blend of hypernetworks and meta-learning techniques in the PINN space in order to build efficient models that can work or adapt across a variety of PDE parameters is of increasing notice and importance. This work is positioned well against similar recent works and provides an intuitive and thought-out approach for handling all of these components together, especially the smart use of hypernetworks in outputting a smaller set of important parameters as a component of the low-rank decomposition.\n\n- The writing is clean and clear and the sections are structured well.\n- The algorithm is explained clearly and it is easy to follow the all the moving components. \n- The research questions problems are clearly laid out and have corresponding answers in the model design.\n- The experiments are robust, with ablations that support each of the research question, and a comprehensive suite of baselines are considered.\n- The appendix is similarly robust, providing both visualizations and analogies to relevant background techniques that enhances understanding. The inclusion of the preliminary experiments that motivated the techniques proposed is additionally welcome.\n- The code is clean and reproducible, given provided environment files and a detailed README with instruction. All data generators are provided as well.\n\nWEAKNESSES:\n- The lack of explanation on the meta-learning inclusion to the framework in the main paper is a bit confusing if a reader isn't coming from a meta-PINN perspective as there are no definitions given in the standard form of meta-train and meta-test environments or details on the context/query sets. It is elaborated on more in the Appendix and it can be pieced together in the work but perhaps a more clear reference to that section in the main work or elaboration on how meta-learning is formulated in this domain may provide more general clarity.\n\nQUESTIONS:\n** **\n\nLIMITATIONS:\nSocietal impact and limitations of the model/experimentation is described appropriately, with further information on future work provided in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for solving PDEs by using a low-rank parameterized NN.\nWhile traditional PINNs are restricted to a single PDE instance, the authors use a hypernetwork to be able to solve PDEs with different parameters. The hypernetwork predicts the weights matrices of a PINN network in a low-rank fashion: each weight matrix is decomposed via SVD, and only the singular values diagonal matrix are modulated by the hypernetwork, while the other two matrices are shared across all the different PDEs. This allows quick inference on unseen PDEs, by learning only the diagonal terms.\nPerformance is evaluated on 1D-convection-reaction-diffusion and 2D-Helmholtz PDEs. It is shown that this low-rank hypernetwork architecture allows to learn on previous failure cases of the vanilla PINN (eg high convection or high reaction). \nThe hypernetwork method shows strong performance, compared to other meta-learning methods (MAML, Reptile), as well as other PINN variants.\n\nSTRENGTHS:\n- The paper is well-written, and provides an extensive set of experiments and ablations showing the relevance of the low-rank decomposition, proposed loss terms and evaluate\n- The performance of the proposed model is very competitive, while also providing much faster inference compared to the other meta-learning algorithms, and generalizing better than the other PINN variants\n\nWEAKNESSES:\n- “low-rank PINNs containing only hundreds of model parameters” (l. 10): the parameter comparison seems unfair, since only the parameters learned during phase 2 are counted (not hypernetwork or basis vectors parameters)\n- The adaptive rank (section K) in appendix is missing\n\nQUESTIONS:\n- \"For higher values (20), it appears that a higher rank is required to achieve a certain level of accuracy, presumably due to numerical reasons.” (l.503): what do you mean by numerical reasons?\n- How does the proposed architecture scale to higher dimensions?\n- How does the Phase 1 computation compare to the PINN variants?\n\nLIMITATIONS:\nThe method is currently limited to generalizing to new PDE parameters only, it would be interesting to see if it can be extended to different initial / boundary conditions", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}