SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Woojin Cho† Kookjin Lee‡* Donsub Rim§ Noseong Park†*
† Yonsei University
† Arizona State University
$\S$  Washington University in St. Louis
snowmoon@yonsei.ac.kr, kookjin.lee@asu.edu,
rim@wustl.edu, noseong@yonsei.ac.kr
In various engineering and applied science applications, repetitive numerical simulations of partial differential equations (PDEs) for varying input parameters are often required (e.g., aircraft shape optimization over many design parameters) and solvers are required to perform rapid execution. In this study, we suggest a path that potentially opens up a possibility for physics-informed neural networks (PINNs), emerging deep-learning-based solvers, to be considered as one such solver. Although PINNs have pioneered a proper integration of deep-learning and scientific computing, they require repetitive time-consuming training of neural networks, which is not suitable for many-query scenarios. To address this issue, we propose lightweight low-rank PINNs containing only hundreds of model parameters and an associated hypernetwork-based meta-learning algorithm, which allow efficient solution approximations for varying PDE input parameters. Moreover, we show that the proposed method is effective in overcoming a challenging issue, known as "failure modes" of PINNs.
Physics-informed neural networks (PINNs) [1] are a particular class of coordinate-based multi-layer perceptrons (MLPs), also known as implicit neural representations (INRs), to numerically approximate solutions of partial differential equations (PDEs). That is, PINNs are taking spatiotemporal coordinates  $(\pmb{x}, t)$  as an input and predict PDE solutions evaluated at the coordinates  $u_{\Theta}(\pmb{x}, t)$  and are trained by minimizing (implicit) PDE residual loss and data matching loss at initial and boundary conditions. PINNs have been successfully applied to many different important applications in computational science and engineering domain: computational fluid dynamics [2, 3], cardiac electrophysiology simulation [4], material science [5], and photonics [6], to name a few.
PINNs are, however, sharing the same weakness with coordinate-based MLPs (or INRs), which hinders the application of PINNs/INRs to more diverse applications; for a new data instance (e.g., a new PDE for PINNs or a new image for INRs), training a new neural network (typically from scratch) is required. Thus, using PINNs to solve PDEs (particularly, in parameterized PDE settings) is usually computationally demanding, and this burden precludes the application of PINNs to important scenarios that involve many queries in nature as these scenarios require the parameterized PDE models to be simulated thousands of times (e.g., design optimization, uncertainty propagation), i.e., requiring PDE solutions  $u(\pmb{x}, t; \pmb{\mu})$  at many PDE parameter settings  $\{\pmb{\mu}^{(i)}\}_{i=1}^{N_{\mu}}$  with very large  $N_{\mu}$ .
To mitigate the above described issue, we propose i) a low-rank structured neural network architecture for PINNs, denoted as low-rank PINNs (LR-PINNs), ii) an efficient rank-revealing training algorithm, which adaptively adjust ranks of LR-PINNs for varying PDE inputs, and iii) a two-phase procedure (offline training/online testing) for handling many-query scenarios. This study is inspired by the observations from the studies of numerical PDE solvers [7-11] stating that numerical solutions of parametric PDEs can be often approximated in a low-rank matrix or tensor format with reduced computational/memory requirements. In particular, the proposed approach adopts the computational formalism used in reduced-order modeling (ROM) [12, 13], one of the most dominant approaches in solving parametric PDEs, which we will further elaborate in Section 3.
In essence, LR-PINNs represent the weight of some internal layers as a low-rank matrix format. Specifically, we employ a singular-value-decomposition (SVD)-like matrix decomposition, i.e., a linear combination of rank-1 matrices: the weight of the  $l$ -th layer is  $W^{l} = \sum_{i=1}^{r} s_{i}^{l} \boldsymbol{u}_{i}^{l} \boldsymbol{v}_{i}^{l^{\top}}$  with the rank  $r = \min(n_{l}, n_{l+1})$ , where  $W^{l} \in \mathbb{R}^{n_{l+1} \times n_{l}}$ ,  $\boldsymbol{u}_{i}^{l} \in \mathbb{R}^{n_{l+1}}$ ,  $\boldsymbol{v}_{i}^{l} \in \mathbb{R}^{n_{l}}$ , and  $s_{i}^{l} \in \mathbb{R}$ . The ranks of the internal layers, however, typically are not known a priori. To address this issue, we devise a novel hypernetwork-based neural network architecture, where the rank-structure depending on the PDE parameters  $\pmb{\mu}$  is learned via training. In short, the proposed architecture consists of i) a lightweight hypernetwork module and ii) a low-rank solution network module; the hypernetwork takes in the PDE parameters and produces the coefficients of the rank-1 series expansion (i.e.,  $\boldsymbol{s}(\boldsymbol{\mu}) = f^{\mathrm{hyper}}(\boldsymbol{\mu})$ ). The low-rank solution network module i) takes in the spatiotemporal coordinates  $(\boldsymbol{x}, t)$ , ii) takes the forward pass through the linear layers with the weights  $W^{l}(\boldsymbol{\mu}) = \sum_{i=1}^{r} s_{i}^{l}(\boldsymbol{\mu}) \boldsymbol{u}_{i}^{l} \boldsymbol{v}_{i}^{l^{\top}}$ , of which  $s_{i}(\boldsymbol{\mu})$  comes from the hypernetwork, and iii) produces the prediction  $u_{\Theta}(\boldsymbol{x}, t; \boldsymbol{\mu})$ . Then, the training is performed via minimizing the PINN loss, i.e., a part of the PINN loss is the PDE residual loss,  $\| \mathcal{R}(u_{\Theta}(x, t; \boldsymbol{\mu}); \boldsymbol{\mu}) \|_2$ , where  $\mathcal{R}(\cdot, \cdot; \boldsymbol{\mu})$  denotes the parameterized PDE residual operator.
We show the efficacy and the efficiency of our proposed method for solving some of the most fundamental parameterized PDEs, called convection-diffusion-reaction equations, and Helmholtz equations. Our contributions include:
1. We employ a low-rank neural network for PINNs after identifying three research challenges to address.
2. We develop a hypernetwork-based framework for solving parameterized PDEs, which computes solutions in a rank-adaptive way for varying PDE parameters.
3. We demonstrate that the proposed method resolves the "failure modes" of PINNs.
4. We also demonstrate that our method outperforms baselines in terms of accuracy and speed.
2 Naive low-rank PINNs
Let us be by formally defining LR-PINNs and attempt to answer relevant research questions. LR-PINNs are a class of PINNs that has hidden fully-connected layers (FC) represented as a low-rank weight matrix. We denote this intermediate layer as LR-FC: the  $l$ -th hidden layer is defined such that
$$
\boldsymbol {h} ^ {l + 1} = \operatorname {L R - F C} ^ {l} (\boldsymbol {h} ^ {l}) \quad \Leftrightarrow \quad \boldsymbol {h} ^ {l + 1} = U _ {r} ^ {l} \left(\Sigma_ {r} ^ {l} \left(V _ {r} ^ {l \top} \boldsymbol {h} ^ {l}\right)\right) + \boldsymbol {b} ^ {l}, \tag {1}
$$
where  $U_r^l \in \mathbb{R}^{n_{l+1} \times r}$  and  $V_r^l \in \mathbb{R}^{n_l \times r}$  denote full column-rank matrices (i.e., rank  $r \ll n_l, n_{l+1}$ ) containing a set of orthogonal basis vectors, and  $\Sigma_r^l \in \mathbb{R}^{r \times r}$  is a diagonal matrix,  $\Sigma_r^l = \operatorname{diag}\left(\boldsymbol{s}_r^l\right)$  with  $\boldsymbol{s}_r^l \in \mathbb{R}^r$ .
Memory efficiency: LR-PINNs with a rank of  $r$  and  $L$  hidden layers require  $O((2n_{l} + 1)rL)$  memory as opposed to  $O(n_{l}^{2}L)$  required by regular PINNs.
Computational efficiency: The forward/backward pass of LR-PINNs can be computed efficiently by utilizing a factored representation of the weights. To simplify the presentation, we describe only the forward pass computation; the forward pass is equivalent to perform three small matrix-vector products (MVPs) in sequence as indicated by the parentheses in Eq. (1).
Challenges: Representing the weights of hidden layers itself is straightforward and indeed has been studied actively in many different fields of deep learning, e.g., NLP [14, 15]. However, those approaches typically assume that there exist pre-trained models and approximate the model weights by running the truncated SVD algorithm. Our approach is different from these approaches in that we
Figure 1: The architecture of Hyper-LR-PINN consisting of i) the hypernetwork generating model parameters (i.e., diagonal elements) of LR-PINN and ii) LR-PINN approximating solutions.
attempt to reveal the ranks of internal layers as the training proceeds, which brings unique challenges. These challenges can be summarized with some research questions, which include: C1) "should we make all parameters learnable (i.e.,  $(U^l,V^l,\Sigma^l))$ ", C2) "how can we determine the ranks of each layer separately, and also adaptively for varying  $\mu$ ?", and C3) "can we utilize a low-rank structure to avoid expensive and repetitive training of PINNs for every single new  $\mu$  instances?". In the following, we address these questions by proposing a novel neural network architecture.
3 Hyper-LR-PINNs: hypernetwork-based meta-learning low-rank PINNs
We propose a novel neural network architecture based on a hypernetwork and an associated training algorithm, which address the predefined challenges. To distinguish them, we hereinafter call LR-PINN with (resp. w/o) our proposed hypernetwork as "Hyper-LR-PINN" (resp. "Naive-LR-PINN").
Design goals: Here we attempt to resolve all challenges by setting up several design goals that are inspired from our domain knowledge in the field (and also from some preliminary results that can be found in Appendix D):
1. build a single set of basis vectors  $U^l$  and  $V^l$ , preferably as orthogonal as possible, that perform well over a range of PDE parameters,
2. build LR-PINNs with an adaptive and layer-wise rank structure that depends on PDE parameters (e.g., a higher rank for a higher convective PDE) and,
3. make only the diagonal elements, denoted  $s_r^l$  in Eq. (1), learnable to achieve high efficiency once a proper set of basis vectors and the rank structure are identified.
Our design largely follows the principles of ROMs, where the expensive computation is offloaded to an offline phase to build a cheap surrogate model that can perform an efficient computation in an online phase for a test set. To make an analogy, consider parameterized dynamical systems (which may arise in semi-discretization of time-dependent PDEs):  $\frac{\mathrm{d}\pmb{u}(t;\pmb{\mu})}{\mathrm{d}t} = \pmb {f}(\pmb {u}(t;\pmb {\mu});\pmb {\mu})$  , where  $\pmb {u}\in \mathbb{R}^N$  . In the offline phase, the method seeks a low-dimensional linear trial basis where the reduced representation of the solution lie on, which is achieved by performing high-fidelity simulations on a set of PDE parameter instances  $\{\pmb{\mu}^{(i)}\}_{i = 1}^{n_{\mathrm{train}}}$  and constructing a trial linear subspace  $\Psi \coloneqq \mathrm{range}(\Psi_p)$  with  $\Psi_p = [\psi_1,\dots ,\psi_p]\in \mathbb{R}^{N\times p}$  from the solution snapshots collected from the high-fidelity simulations. In the online phase, the solutions at a set of test PDE parameter instances  $\{\pmb{\mu}^{(i)}\}_{i = 1}^{n_{\mathrm{test}}}$  are approximated as  $\pmb {u}(t,\pmb {\mu})\approx \Psi_{p}\pmb {c}(t,\pmb {\mu})$  with  $c\in \mathbb{R}^p$  , and a low-dimensional surrogate problem is derived as  $\frac{\mathrm{d}\pmb{c}(t;\pmb{\mu})}{\mathrm{d}t} = \Psi_p^\top \pmb {f}(\Psi_p\pmb {c}(t;\pmb {\mu});\pmb {\mu}) = \hat{\pmb{f}} (\pmb {c}(t;\pmb {\mu});\pmb {\mu})\in \mathbb{R}^p$  , which can be rapidly solved, while not losing too much accuracy. See Appendix C for an illustrative explanation and details of ROMs.
Taking a cue from the ROM principles, we design our model to operate on a common set of basis vectors  $\{U_r^l\}$  and  $\{V_r^l\}$ , which are obtained during the offline phase (analogous to  $\Psi_p$  in ROMs) and update only the diagonal elements  $\{s_r^l\}$  during the online phase (analogous to  $c$  in ROMs). Now, we elaborate our network design and the associated two-phase algorithm. For the connection to the ROM, which explains details on the context/query sets, can be found in Appendix C
3.1 Hypernetwork-based neural network architecture
The proposed framework has two computational paths: a path for the hypernetwork and a path for LR-PINN (Figure 1). The hypernetwork path reads the PDE parameter  $\pmb{\mu}$  and outputs the diagonal elements  $\{s^l\}_{l = 1}^L$  of LR-FCs. The LR-PINN path reads  $(x,t)$  and the output of the hypernetwork, and outputs the approximated solution  $u_{\Theta}$  at  $(x,t;\pmb {\mu})$ , which can be written as follows:
$$
u _ {\Theta} ((\boldsymbol {x}, t); \boldsymbol {\mu}) = u _ {\Theta} ((\boldsymbol {x}, t); f ^ {\text {h y p e r}} (\boldsymbol {\mu})),
$$
where  $f^{\mathrm{hyper}}(\pmb{\mu})$  denotes the hypernetwork such that  $\{s^l (\pmb {\mu})\}_{l = 1}^L = f^{\mathrm{hyper}}(\pmb {\mu})$ . We denote  $\pmb{s}$  as a function of  $\pmb{\mu}$  to make it explicit that it is dependent on  $\pmb{\mu}$ . The internals of LR-PINN can be described as with  $\pmb {h}^0 = [\pmb {x},t]^{\top}$
$$
\boldsymbol {h} ^ {1} = \sigma (W ^ {0} \boldsymbol {h} ^ {0} + \boldsymbol {b} ^ {0}),
$$
$$
\boldsymbol {h} ^ {l + 1} = \sigma (U ^ {l} (\Sigma^ {l} (\boldsymbol {\mu}) (V ^ {l \top} \boldsymbol {h} ^ {l})) + \boldsymbol {b} ^ {l}), l = 1, \dots , L,
$$
$$
u _ {\Theta} ((\boldsymbol {x}, t); \boldsymbol {\mu}) = \sigma \left(W ^ {L + 1} \boldsymbol {h} ^ {L + 1} + \boldsymbol {b} ^ {L + 1}\right),
$$
where  $\Sigma^l (\pmb {\mu}) = \mathrm{diag}(\pmb {s}^l (\pmb {\mu}))$ . The hypernetwork can be described as the following initial embedding layer, where  $\pmb{e}^{0} = \pmb{\mu}$ , followed by an output layer:
$$
\boldsymbol {e} ^ {m} = \sigma (W ^ {\mathrm {e m b}, m} \boldsymbol {e} ^ {m - 1} + \boldsymbol {b} ^ {\mathrm {e m b}, m}), \quad m = 1, \dots , M,
$$
$$
\boldsymbol {s} ^ {l} (\boldsymbol {\mu}) = \operatorname {R e L U} \left(W ^ {\text {h y p e r}, l} \boldsymbol {e} ^ {M} + \boldsymbol {b} ^ {\text {h y p e r}, l}\right), \quad l = 1, \dots , L,
$$
where  $\mathrm{ReLU}$  is employed to automatically truncate the negative values so that the adaptive rank structure for varying PDE parameters can be revealed (i.e., the number of non-zeros  $(\mathrm{NNZs})^2$  in  $s^l (\pmb {\mu})$  varies depending on  $\pmb{\mu}$ ).
3.2 Two-phase training algorithm
Along with the framework, we present the proposed two-phase training algorithm. Phase 1 is for learning the common set of basis vectors and the hypernetwork and Phase 2 is for fine-tuning the network for a specific set of test PDE parameters. Table I shows the sets of model parameters that are being trained in each phase. (See Appendix E for the formal algorithm.)
In Phase 1, we train the hypernetwork and the LR-PINN jointly on a set of collocation points that are collected for varying PDE parameters. Through the computational procedure described in Section 3.1, the approximated solutions at the collocation points are produced  $u_{\Theta}((\boldsymbol{x}_j,t_j);\pmb{\mu}^{(i)})$ . Then, as in regular PINNs, the PDE residual loss and the data matching loss can be computed. The small difference is that the PDE operator,  $\mathcal{F}$ , for the residual loss is also parameterized such that  $\mathcal{F}(u_{\Theta}((\boldsymbol{x}_j,t_j);\pmb{\mu}^{(i)});\pmb{\mu}^{(i)})$ . As we wish to obtain basis vectors that are close to orthogonal, we add the following orthogonality constraint based on the Frobenius norm to the PINN loss [16]:
$$
w _ {1} \| U ^ {l ^ {\mathsf {T}}} U ^ {l} - I \| _ {F} ^ {2} + w _ {2} \| V ^ {l ^ {\mathsf {T}}} V ^ {l} - I \| _ {F} ^ {2}, \tag {2}
$$
where  $w_{1}$  and  $w_{2}$  are penalty weights. (See Appendix J.)
In Phase 2, we continue training LR-PINN for approximating the solutions of a target PDE parameter configuration. We i) fix the weights, the biases, and the set of basis vectors of LR-FC obtained from Phase 1, ii) convert the diagonal elements to a set of learnable parameters after initializing them with the values from the hypernetwork, and iii) detach the hypernetwork. Thus, only the trainable parameters from this point are the set of diagonal elements, first and last linear layers. The hypernetwork-initialized diagonal elements serve as a good starting point in (stochastic) gradient update optimizers (i.e., require less number of epochs). Moreover, significant computational savings can be achieved in the gradient update steps as only the diagonal elements are updated (i.e.,  $\pmb{s}_{l+1} \gets \pmb{s}_l + \eta \nabla_s L$  instead of  $\Theta_{l+1} \gets \Theta_l + \eta \nabla_\Theta L$ ).
We demonstrate that our proposed method significantly outperforms baselines on the 1-dimensional/2-dimensional PDE benchmarks that are known to be very challenging for PINNs to learn [17, 18]. We report the average accuracy and refer readers to Appendix  $\boxed{\mathbf{U}}$  for the std. dev. of accuracy after 3 runs.
2The NNZs is not equal to the rank as the learned basis is only soft-constrained to be orthogonal (Eq. 2). Nevertheless, to avoid notation overloading, we denote NNZs as the rank in the following.
Table 1: Learnable parameters in each phase
Phase 1	{(Ul, Vl, b)}Ll=1, W0, WL+1, b0, bL+1	(LR-PINN),
{Wemb,m, bemb,m}Mm=1, {(Whyper,l, bhyper,l)}Ll=1	(hypernetwork),
Phase 2	{sI}Ll=1, W0, WL+1, b0, bL+1	(LR-PINN)
(a) Exact solution
(b) PINN
(c) Ours
Figure 2: [Convection equation] Solution snapshots for  $\beta = 40$
Baselines for comparison: Along with the vanilla PINN [1], our baselines include several variants. PINN-R [19] denotes a model that adds skip-connections to PINN. PINN-S2S [18] denotes a method that uniformly segments the temporal domain and proceeds by training each segment one by one in a temporal order. PINN-P denotes a method that directly extends PINNs to take  $(\mathbf{x}, t, \boldsymbol{\mu})$  as input and infer  $u_{\theta}(\mathbf{x}, t, \boldsymbol{\mu})$ , i.e.,  $\boldsymbol{\mu}$  is being treated as a coordinate in the parameter domain.
We also apply various meta-learning algorithms to Naive-LR-PINN: model-agnostic meta learning (MAML) [20] and Reptile [21] — recall that Naive-LR-PINN means that LR-PINN without our proposed hypernetwork-based meta-learning and therefore, MAML and Reptile on top of Naive-LR-PINN can be compared to Hyper-LR-PINN. In the parameterized PDE setting, we can define a task,  $\tau^{(i)}$ , as a specific setting of the PDE parameters,  $\pmb{\mu}^{(i)}$ . Both MAML and Reptile seek initial weights of a PINN, which can serve as a good starting point for gradient-based optimizers when a solution of a new unseen PDE parameter setting is sought. See Appendix G for details. For reproducibility, we refer readers to Appendix F including hyperparameter configuration and software/hardware environments.
Evaluation metrics: Given the  $i$ -th PDE parameter instance  $\pmb{\mu}^{(i)}$ , the ground-truth solution evaluated at the set of test collocation points can be defined collectively as  $\pmb{u}^{(i)} = [u(x_1, t_1; \pmb{\mu}^{(i)}), \dots, u(x_N, t_N; \pmb{\mu}^{(i)})]^{\top}$  and likewise for PINNs as  $\pmb{u}_{\theta}^{(i)}$ . Then the absolute error and the relative error can be defined as  $\frac{1}{N} \| \pmb{u}^{(i)} - \pmb{u}_{\theta}^{(i)} \|_1$  and  $\| \pmb{u}^{(i)} - \pmb{u}_{\theta}^{(i)} \|_2 / \| \pmb{u}^{(i)} \|_2$ , respectively. In Appendix  $\boxed{\mathbf{N}}$ , we measure the performance on more metric: max error and explained variance score.
4.1 Benchmark parameterized PDEs
1D PDEs: For our first benchmark parameterized PDEs, we consider the following parameterized convection-diffusion-reaction (CDR) equation:
$$
u _ {t} + \beta u _ {x} - \nu u _ {x x} - \rho u (1 - u) = 0, \quad x \in \Omega , t \in [ 0, T ],
$$
where the equation describes how the state variable  $u$  evolves under convective (the second term), diffusive (the third term), and reactive (the fourth term) phenomena. The triplet,  $\mu = (\beta, \nu, \rho)$ , defines the characteristics of the CDR equations: how strong convective/diffusive/reactive the equation is, respectively. A particular choice of  $\mu$  leads to a realization of a specific type of CDR processes.
There is a set of  $\pmb{\mu}$  which makes training PINNs very challenging, known as "failure modes" [18]: i) convection equations with high convective terms ( $\beta \geq 30$ ) and ii) reaction(-diffusion) equations with high reactive terms ( $\rho \geq 5$ ). We demonstrate that our method does not require specific-PDE-dedicated algorithms (e.g., [18]) while producing comparable/better accuracy in low rank.
3Note that we consider the Fisher's form  $\rho u(1 - u)$  as the reaction term following [18].
(b) The absolute error
Figure 3: [Convection equation] Per epoch averaged train loss and test errors for  $\beta \in [30,40]$  for Phase 1 (meta-learning phase). We refer readers to Appendix O for Phase 2 loss curve.
(c) The relative error
Table 2: [Convection equation] The absolute and relative errors of the solutions of convection equations with  $\beta = \{30,35,40\}$ . We apply the curriculum learning proposed in [18], MAML, and Reptile to Naive-LR-PINN. Therefore, we focus on comparing various Naive-LR-PINN-based enhancements and our Hyper-LR-PINN. See Appendix [N] for other omitted tables.
β	Rank	[w/o] Pre-training	[ w ] Pre-training
Naïve-LR-PINN	Curriculum learning	MAML	Reptile	Hyper-LR-PINN (Full rank)	Hyper-LR-PINN (Adaptive rank)
Abs. err.	Rel. err.	Abs. err.	Rel. err.	Abs. err.	Rel. err.	Abs. err.	Rel. err.	Abs. err.	Rel. err.	Abs. err.	Rel. err.
30	10	0.5617	0.5344	0.4117	0.4098	0.6757	0.6294	0.5893	0.5551
20	0.5501	0.5253	0.4023	0.4005	0.6836	0.6452	0.6144	0.5779	0.0360	0.0379	0.0375	0.0389
30	0.5327	0.5126	0.4233	0.4204	0.5781	0.5451	0.6048	0.5704
40	0.5257	0.5076	0.3746	0.3744	0.5848	0.5515	0.5757	0.5442
50	0.5327	0.5126	0.4152	0.4127	0.5898	0.5562	0.5817	0.5496
35	10	0.5663	0.5357	0.5825	0.5465	0.6663	0.6213	0.5786	0.5446	0.0428	0.0443	0.0448	0.0461
20	0.5675	0.5369	0.6120	0.5673	0.6814	0.6433	0.5971	0.5606
30	0.6081	0.5670	0.5864	0.5503	0.5819	0.5466	0.5866	0.5506
40	0.5477	0.5227	0.5954	0.5548	0.5809	0.5462	0.5773	0.5435
50	0.5449	0.5208	0.6010	0.5619	0.5870	0.5514	0.5731	0.5404
40	10	0.5974	0.5632	0.5978	0.5611	0.6789	0.6446	0.5992	0.5632	0.0603	0.0655	0.0656	0.0722
20	0.5890	0.5563	0.6274	0.5820	0.7008	0.6801	0.6189	0.5853
30	0.6142	0.5724	0.6011	0.5652	0.6072	0.5700	0.6126	0.5810
40	0.5560	0.5293	0.6126	0.5715	0.6149	0.5832	0.6004	0.5638
50	0.6161	0.5855	0.6130	0.5757	0.6146	0.5799	0.6007	0.5645
2D PDEs: As the second set of benchmarks, we consider the 2-dim parameterized Helmholtz equation:
$$
u _ {x x} + u _ {y y} + k ^ {2} u - q (x, y; a _ {1}, a _ {2}) = 0,
$$
where  $q(x,y;a_1,a_2)$  denotes a specific parameterized forcing term and the solution  $u$  can be calculated analytically (See Appendix). As observed in the failure modes of convection equations (high convective terms), certain choices of the parameters in the forcing term, i.e.,  $a_1,a_2$ , make the training of PINNs challenging as the solutions become highly oscillatory.
4.2.1 Performance on the failure mode of the CDR equation
Solution accuracy: As studied in prior work, vanilla PINNs fail to approximate solutions exhibiting either highly oscillatory (due to high convection) or sharp transient (due to high reaction) behaviors. Here, we present the results of convection equations and leave the results on reaction(-diffusion) equations in Appendix N. For convection equations, failures typically occur with high  $\beta$ , e.g.,  $\beta \geq 30$ . Table 2 reports the results of all considered models trained on  $\beta \in [30,40]$  and essentially shows that Hyper-LR-PINN (Figure 2(c)) is the only low-rank method that can resolve the failure-mode and there are only marginal decreases in accuracy compared to Hyper-LR-PINN in full rank. For reaction(-diffusion) equations, we observe similar results, Hyper-LR-PINN outperforms the
4 Initial condition:  $1 + \sin (x)$  and boundary condition: periodic
Following [18], we consider only the coefficients in the natural numbers (e.g.,  $\beta \in \mathbb{N}_{+}$ ) and [30,40] indicates the set  $\{30, 31, \ldots, 40\}$ .