SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Comparisons of model size
Model	Naïve-LR-PINN	Ours	PINN
Rank	10	20	30	40	50	Adaptive	-
# Parameters	381	411	441	471	501	~351	10,401
(a)  $\beta \in [30,40]$
(b)  $\beta \in [1,20]$
(c)  $\beta \in [1,20]$
Figure 4: Adaptive rank on convection equation (the left and the middle panels). The magnitude of the learned diagonal elements  $s^2$  of the second hidden layer for varying  $\beta \in [1,20]$  (the right panel).
best baseline by more than an order of magnitude (See Appendix N). Moreover, we report that Hyper-LR-PINN outperforms in most cases to the baselines that operate in "full-rank": meta-learning methods (MAML, Reptile), PINN-S2S, and PINN-P (Appendix M).
Loss curves: Figure 3 depicts two curves of train loss and test errors as the meta-learning algorithms proceed (MAML, Reptile, and ours). As opposed to the optimization-based meta-learning algorithms, which tend to learn meta-initial weights that perform well "on average" over randomly sampled training tasks, our hypernetwork-based method minimizes the loss for each individual task simultaneously. Loss curves for Phase 2 are reported in Appendix  $\boxed{0}$ , which essentially show that the baseline meta-learners do not provide good initializations.
Rank structure: Table3 compares the number of trainable model parameters for Naive-LR-PINN, PINN, and our method. In our model, each hidden layer has a different rank structure, leading to 351 trainable parameters, which is about  $(\times 30)$  smaller than that of the vanilla PINN, Cf. merely decreasing the model size of PINNs leads to poor performance (Appendix S). Figure4(a) shows how the model adaptively learns the rank structure for varying values of  $\beta$ . We observe that, with the low-rank format, Hyper-LR-PINN provides more than  $(\times 4)$  speed up in training time compared to the vanilla PINN. More comparisons on this aspect will be presented in the general CDR settings.
Ablation on fixed or learnable basis vectors: We compare the performance of the cases, where the basis vectors  $\{U_r^l,V_r^l\}$  are trainable. We observe that the fixed basis vectors lead to an order of magnitude more accurate predictions as well as faster convergence (See Appendix Q for plots).
Ablation on orthogonality constraint: We compare the performances of the cases, where the proposed model is trained without the orthogonality constraint, Eq. (2). Experimental results show that the orthogonality constraint is essential in achieve good performance (See Appendix 1).
Many-query settings (inter/extra-polation in the PDE parameter domain): Next, we test our algorithm on the many-query scenario, where we train the proposed framework with the 11 training PDE parameters  $\beta \in [30,40]$  (Phase 1), freeze the hypernetwork and the learned basis, and retrain only the diagonal elements for 150 unseen test PDE parameters (Phase 2). Figure ⑤ depicts the absolute and relative errors of the solutions, which are as accurate as the ones on the training PDE parameters, in particular, for  $\beta < 30$  (i.e., extrapolation). Although the model produces an increasing error for  $\beta > 40$ , the relative error is still around  $\sim 10\%$ , which cannot be achievable by vanilla PINNs. The right chart of Figure ⑤ shows the number of trainable model parameters, which is determined by the rank (i.e., the output of the hypernetwork). As PINNs fail to produce any reasonable approximation in this  $\beta$  regime, we make comparisons in the general CDR settings below.
(a) The absolute error
(b) The relative error
Figure 5: Multi-query scenario: the absolute/relative errors of the predicted solutions at unseen test parameters (left and middle), and the number of trainable model parameters (right).
(c) Model size
Table 4: The relative errors of the solutions of parameterized PDEs with  $\beta, \nu, \rho \in [1,5]$ . The initial condition of each equation is Gaussian distribution  $N(\pi, (\pi/2)^2)$ .
PDE type	No pre-training	Meta-learning
PINN	PINN-P	PINN-S2S	MAML	Reptile	Ours
Convection	0.0327	0.0217	0.2160	0.1036	0.0347	0.0085
Reaction	0.3907	0.2024	0.5907	0.0057	0.0064	0.0045
Conv-Diff-Reac	0.2210	0.2308	0.5983	0.0144	0.0701	0.0329
4.2.2 Performance on the general case of the CDR equation
Solution accuracy: Next, we present the performance of the proposed method on solving the benchmark PDEs (not only in the failure mode but) in all available settings for  $\beta, \nu, \rho$ . Due to space reasons, we present only a subset of our results and leave the detailed results in Appendix M. Table 4 shows that the proposed Hyper-LR-PINN mostly outperforms the baselines that operate in full rank.
Some observations on learned low-rank structures: Again, the proposed method learns rank structures that are adaptive to the PDE parameter (Figure 4(b)). The learned rank structures vary more dynamically ( $r$  from 20 to 34) as the PDE parameter range [1,20] is larger compared to [30,40].
Figure 4(c) visualizes the magnitude of the learned diagonal elements  $s^2$  in the second hidden layer. The horizontal axis indicates the index of the element in  $s^2$ , which is sorted in descending order for analysis, and the vertical axis indicates the  $\beta$  value. The depicted result agrees with our intuitions in many aspects: i) the values decay fast (indicating there exist low-rank structures), ii) the value of each element  $s_i^2$  either increases or decreases gradually as we vary  $\beta$ , and iii) for higher  $\beta$ , higher frequency basis vectors are required to capture more oscillatory behavior exhibited in the solutions, which leads to higher ranks. We report the information for other layers and other problems in Appendix L.
Computational cost: Here, we compare the computational cost of Hyper-LR-PINN and PINN in a many-query scenario. We train Hyper-LR-PINN for  $\beta \in [1,20]$  with interval 1 (Phase 1) and perform Phase 2 on  $\beta \in [1,20]$  with interval 0.5 (including unseen test PDE parameters). We record the number of epochs required for PINN and Hyper-LR-PINN to reach an L2 absolute error of less than 0.05. Figure 6 reports the number of required epochs for each  $\beta$ , showing that the number of epochs required for Hyper-LR-PINN slowly increases while that for PINNs increases rapidly; for most cases, Hyper-LR-PINN reaches the target (i.e., 0.05) in only one epoch. We also emphasize that Hyper-LR-PINN in Phase 2 requires much less FLOPS in each epoch (See Section 3.2).
Figure 6: Computational cost in epochs
4.2.3 Performance on the 2D Helmholtz equation
Now we report experimental results on 2D Helmholtz equations for varying  $a_1$  and  $a_2$ . Under the same condition in Table 2, we train Hyper-LR-PINN for  $a \in [2,3]$  with interval 0.1 ( $a = a_1 = a_2$ ), and compare the performance with PINN. Surprisingly, Hyper-LR-PINN approximates the solution
(a) PINN (Abs.err.  $= 0.7403$
(b) Ours (Abs.err.  $= 0.0285$
(c) Exact solution
Figure 7: [2D-Helmholtz equation] Solution snapshots for  $a = 2.5$
with high precision in only 10 epochs in Phase 2, while PINN struggles to find accurate solutions over 2,000 epochs (Figure 7). We show more visualizations and detailed results in Appendix T.
Meta-learning of PINNs and INRs: HyperPINNs [22] share some commonalities with our proposed method in that they generate model parameters of PINNs via hypernetwork [23]. However, hyperPINNs can only generate full-rank weights and do not have capabilities to handle parameterized PDEs. Another relevant work is optimization-based meta-learning algorithms; representatively MAML [20] and Reptile [21]. In the area of INRs, meta-learning methods via MAML and Reptile for obtaining initial weights for INRs have been studied in [24]. In [25, 26], meta-learning methods, which are based on MAML, have been further extended to obtain sparse representation of INRs.
Low-rank formats in neural networks: In natural language processing, the models being employed (e.g., Bert) typically have hundreds of millions of model parameters and, thus, making the computation efficient during the inference is one of the imminent issues. As a remedy, approximating layers in low-rank via truncated SVD has been studied [14, 15]. Modeling layers in low-rank in general has been studied for MLPs [27-30] and convolutional neural network architectures [31]. In the context of PINNs or INRs, there is no low-rank format has been investigated. The work that is the closest to our work is SVD-PINNs [32], which represents the hidden layers in the factored form as in Eq. (1), but always in full-rank.
PINNs and their variants: There have been numerous sequels improving PINNs [1] in many different aspects. One of the main issues is the multi-term objectives; in [19], a special gradient-based optimizer that balances multiple objective terms has been proposed, and in [33], a network architecture that enforce boundary conditions by design has been studied. Another main issue is that training PINNs often fails on certain classes of PDEs (e.g., fail to capture sharp transition in solutions over the spatial/temporal domains). This is due to the spectral bias [34] and, as a remedy, in [18], training approaches that gradually increase the level of difficulties in training PINNs have been proposed. In [35], a Lagrangian-type reformulation of PINNs is proposed for convection-dominated PDEs. There are also other types of improvements including Bayesian-version of PINNs [36] and PINNs that enforce conservation laws [37]. All these methods, however, require training from scratch when the solution of a new PDE is sought.
In this paper, we propose a low-rank formatted physics-informed neural networks (PINNs) and a hypernetwork-based meta-learning algorithm to solve parameterized partial differential equations (PDEs). Our two-phase method learns a common set of basis vectors and adaptive rank structure for varying PDE parameters in Phase 1 and approximate the solutions for unseen PDE parameters by updating only the coefficients of the learned basis vectors in Phase 2. From the extensive numerical experiments, we have demonstrated that the proposed method outperforms baselines in terms of accuracy and computational/memory efficiency and does not suffer from the failure modes for various parameterized PDEs.
Limitations: Our Hyper-LR-PINN primarily concentrates on the cases where the PDE parameters are the coefficients of PDE terms. As a future study, we plan to extend our framework to more general settings, where the PDE parameters define initial/boundary conditions, or the shape of domains. See Appendix A for more details.
Broader impacts: Hyper-LR-PINN can solve many equations even for PINN's failure modes with high precision. However, despite its small errors, one should exercise caution when applying it to real-world critical applications.
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program ay Yonsei University,  $10\%$ ), and (No.2022-0-00857, Development of AI/databaseq financial/economic digital twin platform,  $90\%$ ).
[1] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[2] Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360:112789, 2020.
[3] XIA Yang, Suhaib Zafar, J-X Wang, and Heng Xiao. Predictive large-eddy-simulation wall modeling via physics-informed neural networks. Physical Review Fluids, 4(3):034602, 2019.
[4] Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
[5] Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using physics-informed neural networks. Science advances, 8(7):eabk0644, 2022.
[6] Wei Ma, Zhaocheng Liu, Zhaxylyk A Kudyshev, Alexandra Boltasseva, Wenshan Cai, and Yongmin Liu. Deep learning for the design of photonic structures. Nature Photonics, 15(2):77-90, 2021.
[7] Daniel Kressner and Christine Tobler. Low-rank tensor Krylov subspace methods for parametrized linear systems. SIAM Journal on Matrix Analysis and Applications, 32(4):1288-1316, 2011.
[8] Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. Gamm-Mitteilungen, 36(1):53-78, 2013.
[9] Kookjin Lee and Howard C Elman. A preconditioned low-rank projection method with a rank-reduction scheme for stochastic partial differential equations. SIAM Journal on Scientific Computing, 39(5):S828-S850, 2017.
[10] Markus Bachmayr, Albert Cohen, and Wolfgang Dahmen. Parametric PDEs: Sparse or low-rank approximations? IMA Journal of Numerical Analysis, 38(4):1661-1708, 2018.
[11] Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
[12] Philip Holmes, John L Lumley, Gahl Berkooz, and Clarence W Rowley. Turbulence, coherent structures, dynamical systems and symmetry. Cambridge university press, 2012.
[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[48] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.