Adaptive Physics-informed Neural Networks: A Survey
Authors: Edgar Torres
Mathias Niepert
Venue: Accepted by TMLR ()

Abstract: Physics-informed neural networks (PINNs) have emerged as a promising approach for solving partial differential equations (PDEs) using neural networks, particularly in data-scarce scenarios due to their unsupervised training capability. However, a key limitation is the
need for re-optimization with each change in PDE parameters, similar to the challenge in traditional numerical methods where each system of equations corresponds to a specific PDE instance. This characteristic poses a barrier to the widespread adoption of PINNs
across scientific and engineering applications. This survey explores research addressing this limitation through transfer learning and meta-learning, synthesizing insights to establish a foundation for efficient data generation strategies tailored to PINNs. These methods can potentially improve PINNs’ training efficiency, enabling quicker adaptation to new PDEs with fewer data and computational demands. While numerical methods directly solve systems of equations to derive solutions, neural networks implicitly learn solutions by adjusting their parameters. One notable advantage of neural networks lies in their capacity to abstract away from specific problem domains, enabling them to retain, discard, or adapt learned representations to efficiently address similar problems. By understanding how these techniques can be applied to PINNs, this survey seeks to identify promising directions for future research to enable the widespread adoption of PINNs across a wide range of scientific and engineering applications.

Published in Transactions on Machine Learning Research (03/2025)
Adaptive Physics-informed Neural Networks: A Survey
Edgar Torres
Institute for Artificial Intelligence University of Stutigort
Jonathan Schiefer
hww.:/ieeeeee@ddbbbbb.b.ooooo com
Robert Bosch GmbHI, Stuttgart
Mathias Niepert
mathias. sigtiiaannieeeertttiiiiiiittttto
Institute for Artificial Intelligence University of Stuttgert
Reviewed on OpenReview: https: // openreview. net/ forum? id=uz5P1kbt6)
//
Abstract
Physics-informed neural networks (PINNs) have emerged as a promising approach solv- ing partial differential equations (PDEs) using neural networks, particularly in data-scarce scenarios, due to their unsupervised training capability. However, limitations related to con- vergence and the need for re-optimization with each change in PDE parameters hinder their widespread adoption across scientific and engineering applications. This survey reviews ex- isting research that addresses these limitations through transfer learning and meta-learning. The covered methods improve the training efficiency, allowing faster adaptation to new PDEs with fewer data and computational resources. While traditional numerical methods solve systems of differential equations directly, neural networks learn solutions implicitly by adjusting their parameters. One notable advantage of neural networks is their ability to abstract away from specific problem domains, allowing them to retain, discard, adapt learned representations to efficiently address similar problems. By exploring the application of these techniques to PINNs, this survey identifies promising directions for future rescarch to facilitate the broader adoption of PINNs in wide range of scientific and engineering applications.
1 Introduction
Advances in machine learning have led to important applications in various fields, such computer vision (enabling technologies like self-driving cars), natural language processing (powering intelligent agents and chatbots), and image generation (facilitating media creation). Motivated by this success, there has been growing interest in developing Machine Learning (ML) solutions to solve problems in science and engineer- ing. Unlike other fields where data is abundant or casily obtained, however, science and engineering often face data scarcity due the high costs associated with generating data. through expensive experiments or simulations. Therefore, to facilitate the development of ML approaches in these disciplines, AI methods that are data-efficient and compulationally elficient need to be created. To this end, other domains have tackled similar problems with techniques such as transfer learning, meta-learning, and few-shot learning, indicating significant potential for applying these techniques in the context of science and engincering.
One specific application in science and engineering where these elficient ML models can be particularly beneficial to determine the approximate solutions of PDEs. PDEs are fundamental for modeling and describing natural phenomena in various scientific and engincering domains. Traditionally, these equations are solved numerically, which can become prohibitively expensive, especially when dealing with nonlinear and high-dimensional problems (Han et. al. 2018). This challenge limits their application in areas where

---

Published in Transactions on Machine Learning Research (03/2025)
fast evaluation of PDE is required. Recognizing this challenge, neural networks have been explored potential solution, offering advantages in effectively modeling complex nonlinearities (Raissi et al. 2019: Khoo et al. 2021 Sirignano & Spiliopoulos/ 2018 Cuomo et al., 2022), presenting the potential for faster evaluation compared to classical iterative solvers, as well as offering mesh-free solutions not constrained to computational grids (Jiang al.
2023 et al. 2020: Raissi el. al. 2019 Cuomo el. al. 2022 Moreover,
machine Icarning techniques provide an approach to solving inverse problems, where the goal is to infer unknown parameters or initial/boundang conditions from observed data, a task challenging for numerical methods {Arridee et al. 2019 Cai et al. 2021). In addition, machine learning implementations than numerical mothods, allowing faster development and easy maintenance (Cai et al. 2021).
simpler
ML methods for approximating PDE solutions can broadly be categorized into two types: neural surrogates and neural PDE solver: Neural surrogates, inchuding physics-guided neural networks (Faroughi et al. 2023) and neural operators, function by training neural networks to predict data generated from numerical solvers. The most popular among these are neural operators, which approximate nonlincar mappings be- (ween infiittediimensional function spaces using datasets of inpul-oulput pairs from solvers observations. Examples include the Fourier Neural Operator (Li et al.. 2020)) and DeepONet (Lu et al., 20019). On the other hand, neural PDE solvers directly incorporate physical laws by embedding the governing equations into the learning process. A key example is PINNs (Raissi et al. 2019), which approximate solutions by minimizing the residuals the governing equations, the initial conditions, and the boundary conditions. Figure 1 illustrates the relationship between data requirements and scientific knowledge between different methods.
Considering the challenges associated with data quisition in the scientific and engineering domains, this study suggests the use of PINNs to solve such problems. Neural operators typically require large datasets, often dcrived from costly simulations, and do not explicitly incorporate governing physics equations, which can lead generalization prob- lems and physical inconsistencies outside the train- ing data distributions (Négiar et al. 2022). In con- trast, PINNs integrate governing equations directly into the training process, ensuring that the solutions adhere to the underlying physics while reducing the reliance on pre-existing datasets, making them par- ticularly effective for data-scarce applications NN6- giar al. 2022).
Traditional PDF Solvers
Scientific Knowledge
Nenral PDE Solvers
Neural Surrogates
Nevertheless, PINNs have some known limitations. They can struggle with convergence, particularly for complex or high-dimensional physics problems (Négiar et al.) 2022), resulting in long training times. The computation of residuals, which requires derivative evaluation by automatic differentiation, becomes computationally expensive for PDEs with higher-order derivatives. Additionally, PINNs con- vergence is sensitive to hyperparamcters. Further- more, PINNs are typically trained on per-PDE instance basis, meaning they can only solve one specific problem at a time and must be re-trained from scratch for each change in parameters. These drawbacks hinder their adoption in diverse tasks that involve fast evaluations of different PDEs.
Data Quautity
High
Figure 1: Data and scientific knowledge requirements for different modeling approaches.
AA surrogate model be thought of "regrussion" set of data, where the data is set of input-output parings obtained by evaluating black-box model of complex system [ason & Cremaschi (2014); Caballero & Crossmann (0088. In contrast, solver is algorithu method used to find solution mathematical model. While some authors the terms "Neural Surrogates" and "Neural PDE Solvers" Interchangeably, this work makes distinction highlight the specific requirements for obtaining the solution PDF.
2

---

Published in Transactions on Machine Learning Research (03/2025)
To address these limitations, this survey explores the integration of advanced ML techniques, such as transfer learning and meta-learning, into PINNs to maximize knowledge reuse, reduce adaptation time, and mini- mize data requirements. In addition, these methods show potential in addressing some of the convergence challenges associated with PINNs. This survey highlights the idea of efficient model adaptivity for PINNs and its potential facilitate broader adoption in real-world applications where data are searce and fast. evaluation essential.
The key contributions of this work include:
An introductory overview PINNs, highlighting their connections to traditional mumerical methods for solving PDEs and more traditional techniques, such as reduced-order modeling (ROM), that reuse solution data across similar PDE problem instances.
2. A review of recent advances for PINNs, focusing techniques such as transfer learning and meta- learning to improve model adaptivity and efficiency.
Identification potential metrics and benchmarks to assess the adaptivity of the methods.
4. Future research directions and potential applications of adaptive PINNs across various domains.
The paper is structured as follows. First, section introduced key concepts and terminology. Section provides an overview of how transfer learning and meta-learning enhance the adaptivity of PINNs. Section examines benchmarking methodologies and metrics assess adaptation efficiency. Section @explores real-
3
world applications, discusses limitations and areas for improvement, and outlines future research directions. Finally, Section presents the conclusions.
2 Background
2.1 Initial Boundary Value Problem
In science and enginecring, a problem is often framed as an Initial Boundary Value Problem (IBVP), which encompasses a wide range of phenomena. An IBVP is typically represented as:
Mu(x,,t;1)) f(x)
YreΩ, EE [to, T). V 80 t [to, T]. Vze Ω t= to.
Blu(x,t)] g(x)
(1)
I[((I,t) h(x)
where, A represents the differential operator acting on the function u, which depends parameters denoted by μ. [(x) represents the source term defined in the domain Ω. The operator B imposes the boundary conditions g(x) on u at the boundary 0Q of the domain. Lastly, I sets the initial conditions h(x) for the function u(x, to); representing the initial state of u within Ω at the initial time to- The diffcrential operator Ou du 8'y
N can take the form Miu(z, t) F(x, u, Bt Ba 8g ...)...), where F is the dynamics of the system. The objective in solving an IBVP is to find the function u(r, t; μ)) that satisfies the differential equation, the boundary conditions, and the initial conditions simultaneously.
given function that describes
Three methods are often used to solve such IBVP problems mumerically: the Finite Element Method, the Finite Difference Method, and the Finite Volume Method. While these methods differ in their mathematical formulations and approaches, they all discretize the domain into smaller subdomains (elements, cells, or grid points) and perform local approximations to obtain the global solution.
3

---

Published in Transactions on Machine Learning Research (03/2025)
Inputs
Tidden Layers
Solution
Autograd
Losses
Physics-informed
Optimization Operation
Neural Network
pdd
Ldata
No
((0)
update
Yes
End
Figure 2: Schematic representation of PINNs:
The network is optimized by minimizing composite loss function that combincs regression loss from obscrved data, PDE residuals at collocation points, and bound- ary rrnitintal condition losses.
2.2 Physics- Informee Neural Networks
Physics-informes Neural Networks approximate the solution u by using neural network and incorporating IBVP information directly into the training process. Although various methodologies exist, we focus on one of the most common approaches, which involves incorporating the residual the IBVP into the loss function
Raissi al. (2019).
Considering the definition of the IBVP 11, the equations be reformulated in terms of their residuals. These residuals computed at collocation points within the corresponding domain (2, 82, or [to, T)), sampled at discrete locations denoted NPDE, NBC, and Nic-
Nluu(1 ti p)] f(x) TPDE
V (IDE, NPDE tE (PDE, NBc LE {4))
Blue (x, 1)] - g(x) = Yne
V V (a
(2)
[I, t)] h(z) = rIC
VE [o
Mi. t to-
Here, the derivatives of the differential operator M[]]] are computed using automatic differentiation.
The loss lunction is defined as weighted sum of the individual loss terms, where each term is given by C() = MSE(T(), with the placeholder (.) representing the PDE residuals, the boundary condition (BC) or the initial condition (IC). The weights WPDE, WBC, and wie balance the contribution of cach term of the loss
L(uo) WTDE LLDE WBC Tac + Wic LAc-
By optimizing the network parameters with respect to the loss funetion, the network ny aims to learn solution u that approximates the true solution. If partial observation data are available, such from experiments or sensors, an additional regression loss term can be incorporated into Equation B
2.3 Weighted Residuals: Collocation Method
To highlight the similarities between PINNs and numerical methods, this paper compares the collocation method with the PINN approach. For simplicity, steady 1-D case will be considered. The residual of an
TTe terms partial observation data, ground truth data, and sensor data used interchangeahly throughout this work.
4

---

Published in Transactions on Machine Learning Research (03/2025)
IBVP is formulated as:
Mu(a;;]] f(x)= = R(x)
v
(4)
An approximate solution ū is devised in such a form that it is possible to approximate a wide range of functions:
N
((x) = Σ a +,(x,,
(5)
where, given good choice of basis Φ, the task is to find the expansion cocfficients such that the residual is minimized. Since the problem is defined over continuous domain, minimizing the residual requires integrating it over the domain. To account for the spatial variation of the residual, the residual is weighted by w(x), leading to the weighted residual formulation:
[ w(x) R(x) de 0.
(6)
Here, wxx) a weighting (or test) function chosen to enforce the orthogonality of the residual R(z). Common choices include piccewise polynomials, as in finite clement methods, or Dirac delta functions, as used in the method described here.
The collocation
method is special case of the weighted residual formulation (6) where w(x) is chosen as Dirac delta function: w(x) = 8(x æz). This allows direct evaluation of the residual at specific locations. A system of equations can be constructed to approximate the solution by selectively minimizing the residual at these points. The basis functions dx(x) are selected based on the requirements of the problem and play a crucial role in determining the precision and effectiveness of the approximation. Typically, they are chosen to satisfy the boundary conditions and ensure linear independence. These basis functions should possess properties that allow them to accurately capture the behavior of the solution within the problem domain.
2.4 Reduced Order Modeling: numerical approach for reusing information
In the pursuit of efficient. model adaptation, previously obtained solutions to infer new similar solutions, thus reducing computational demands. Reduced order modeling (ROM) encompasses class of numerical techniques that aim to construct simplified version of the original model by reducing its computational complexity. This is achieved by constructing a low-dimensional approximation that captures the essential behavior of the high-fidelity model or simulation but with significantly fewer degrees of freedom. The key objective of ROM is to enable efficient adaptation to new scenarios by leveraging known information from existing simulations, experimental data, or solutions to similar problems.
valuable to explore numerical methods that leverage
One such approach is the Galerkin method combined with Proper Orthogonal Decomposition (POD), often referred to as the Galerkin-POO method. This approach uses the known information from existing solutions (snapshots) to construct a reduced basis consisting of POD basis functions that capture the essential dynamics of the system in various scenarios. This diverse set of snapshots, obtained from multiple tasks or parameter configurations, encapsulates the shared knowledge and dominant features of the system's 's behavior. By performing Proper Orthogonal Decomposition (POD) on these snapshots, the method extracts the dominant POD basis functions that serve as a compact representation of the solution manifold.
Considering the IBVP problem defined in Section the goal of the Galerkin-POD method is to find approximate solution ū(æ, t; 1.) expressed linear combination of the extracted POD basis functions (i(z):
N Mx,1;00 Σ as(l; () (1(x),
where a,(t; p) are the time-dependent modal coefficients and N is the number of retained POD basis functions.
The reduced basis is then used project the governing equations onto a reduced subspace, yielding a reduced system of equations for the modal coefficients. Consequently, when faced with new task
5

---

Published in Transactions on Machine Learning Research (03/2025)
scenario, the Galerkin-POD method can efficiently adapt the solution by solving this reduced system. By reducing the dimensionality, the resulting reduced-order model becomes computationally less expensive to solve or simulate while maintaining an acceptable level of accuracy.
2.5 Relationship with PINNs
Both PINNs and the collocation method leverage residual information to guide the approximated solution toward the ground-truth solution of the governing equations. However, they differ in their approach to constructing the solution ansatz. PINNs exploit the universal approximation theorem, using neural networks as the ansatz solution, with the ability to capture local behavior and sharp gradients dependent on the choice of the activation function. In contrast, the collocation method builds the solution as a linear combination of linearly independent basis functions and expansion coefficients. The selection of basis functions, whether global or piecewise, involves trade-off between accuracy and computational efficiency. Piecowise basis functions, with their compaci support and ability to capture local behavior, can provide higher accuracy for problems with localized features or complex geometries, but at the cost of increased computational complexity and larger systems of equations. Global basis functions, on the other hand, are generally smoother and require fewer basis functions, leading to smaller systems and simpler implementation, but may struggle to accurately represent sharp gradients or complex geometries.
The POD-Galerkin method takes different approach by building a global basis that generalizes to several PDE instances, resulting in fewer equations to solve. By projecting the governing equations onto the reduced space spanned by the global basis, the POD-Galerkin method transmits information from other solutions through the basis functions, enabling efficient and accurate approximations for range of PDE instances.
It is worth noting that the linear combination basis functions in the collocation and POD-Galerkin methods provides a structured and lower-dimensional representation of the solution space, improving computational efficiency. However, this structured approach inherently limits expressibility compared to the more flexible function approximation enabled by PINNs under the universal approximation theorem. This trade-off has motivated recent efforts to frame PINNs in similar linear combination form to balance efficiency and
expressibility (Chen & Koohy 2024
Desai et al. 2021 Peng ct al. 2020 Penwarden et al. 2023 Bischof &
Kraus 2022
2.6 Efficient Model Adaptivity
Efficient model adaptivity refers to the ability of a machine learning model to efficiently and effectively adjust to new, previously unseen tasks using knowledge gained from previous tasks. Given a model pre- trained on one or multiple source tasks T, C T, the goal is to adapt this model to an arbitrary novel target task sampled from l ~ Tj or several larget tasks T. C T where TT, = 0. Tt assumed that all tasks from I' share some common characteristics. In the context of PINNs, a model is typically trained per task, where each task corresponds to an instance of the IBVP, each subject to different parameters μ4, which can be a material property, boundary condition, or initial condition. Figure 3 illlustrates two examples of IBVP. The first example corresponds to the 2D heat equation, where the task-defining parameter is the diffusivity coefficient. The second example corresponds to the Burgers' equation, where the task is defined by its initial condition.
Key aspects that affect efficient model adaptivity include computational efficiency and data efficiency. Com- putational efficiency refers to the model's ability to adapt quickly with minimal computational resources. It encompasses the speed of adaptation, the amount of processing power required, and the overall time needed to adjust the model for new tasks. Computational efficiency is influenced by several factors, primarily the number of model parameters, the complexity of the model, and the optimization steps required for adapta- tion. These elements directly impact the overall training time and computational resources needed. On the other hand, data efficiency refers to how effectively a model can learn from limited data samples. This data can encompass various types: collocation points (evaluation points), partial observations, or the pro-training tasks needed for generalization. This work surveys the application of transfer learning and meta- neta-lcarning physics-informed neural networks to achieve efficient model adaptation.
6

---

Published in Transactions on Machine Learning Research (03/2025)
IBVP Seoup
Dillwiiity:
DiEusiolty: 6.5
Dillusvity:
100
120
a) Heat: Equation
bj Burgens' Equacion
Figure 3: Illustration of IBVP as tasks: a) Heat equation tasks with varying material properties. b) Burgers'
equation tasks with different initial conditions (adapted from Takamoto al. (2022
2.7 Transfer Learning and Darameter-fffcient Fine- Munn-Tunng
Transfer learning is a machine learning approach that transfers knowledge gained from a source domain different but related target domain. The fundamental principle is to leverage a model pre-trained on a source task or dataset and adapt its learned representations accelcrate the learning process and improve generalization performance compared to training from scratch by fine-tuning the pre-trained model on the targel data. This technique is particularly beneficial when the target task has limited labeled data.
new target task. Transfer learning aims
Parameter-officion fine-tuning (PEFT) an advancement in transfer learning that aims to make the fine- luning process even more efficient and scalable. Unlike traditional fine-tuning, which updates all the pa- rameters of the pre-trained model, PEFT introduces small number of trainable parameters that modulate the pre-trained model's behavior. These additional parameters are trained to adapt the pre-trained model to the target task, while most of the original model parameters remain frozen during fine-tuning. Although PEFT was developed for large-scale models, such as large language models, use the term to describe the selective fine-tuning of smaller models, such as those used in PINNs. This interpretation focuses on the principle updating only a subset of parameters to achieve efficient adaptation, regardless of model size.
A Brief History of Transfer Learning
Research in transfer learning dates back to the 1970s and 1980s, as described in the work of Bozinovski (2020) Pratt et al. (1991) conducted pioneering studies that explored to what extent neural networks be "recycled," coining this the "transfer problem.' Sharkey & Sharkey (1993) focused using prior knowledge to improve the performance of new tasks and on understanding when knowledge can be transferred weew networks. Sharkey's work suggests that transfer learning has strong roots in psychology. In 1995, the fundamental motivation for transfer learning was discussed in the NIPS-95 workshop on 'Lcarning Learn' by Baxter et al. (1995). as referenced by Pan Yang (2009). The first comprehensive survey on transfer learning was published by Pan & Yang (2009). More recently, Zhhaag et al. (2020) offered extensive survey covering more than 40 representative transfer learning approaches from both the data and the model perspective.

---

Published in Transactions on Machine Learning Research (03/2025)
2.8 Meta-learning
Meta-learning field in machine learning that encompasses the notion of "learning to learn. The core idea is to leverage external architecture algorithm beyond single-task learning models, which can capture the relationships and shared knowledge across different tasks. By learning the correlations between various tasks during meta-training phase, this external meta-learner can then modify, e.g., the structure of the base learning algorithms, their hyperparameters, and/or the model architectures. This allows the meta-learner to adapt and transfer the acquired meta-knowledgo to new, unscen tasks more efficiently, facilitating rapid learning and generalization. This distinguishes itself from traditional transfer learning, which focuses on adapting between source and target domain. Meta-learning aims to extract task-general knowledge from distribution of tasks, enabling systematic adaptation any task within that distribution.
A Brief History of Meta-learning
The foundations of meta-learning in machine learning can be traced back to several seminal works. For instance, Schmidhuber (1987) introduced the concept of "self-referential learning,' technique where network receives its own weights as inputs and predicts new updates for such weight: Bengio et al. (1900) hypothesized that it is possible to learn algorithms for synaptic learning rules and that these rules could be constrained such that the resulting neural networks can solve complex AI tasks. Later, |Hochreiter et al. (2001) demonstrated that gradient-based optimization can be leveraged to automatically discover effective algorithms tailored to specific tasks, such as time scries forecasting. Expanding on these ideas, Finn al. (2017) rotuudee Model-Agnoste Meta-Learning (MAML), technique designed to enable fast adaptation bi-level optimization: in the inner loop, task-specific parameters are updated through a few gradient steps; in the outer loop, the initialization itself is optimized to minimize the loss across multiple tasks. This approach ensures that the learned initialization is well-suited for rapid adaptation. However, MAML's reliance on second-order derivatives the outer-loop optimization introduces significant computational overhead. To overcome this limitation, Nichol al. (2018) introduced Reptile, a sim- plified first-order approximation to MAML that avoids second-order derivatives. Instead of explicitly computing these gradients, Reptile updates the initialization by directly using the difference between task-specific parameters after inmer-loop updates. This approach retains much of MAML's effec- tiveness while being computationally efficient. These foundational developments have significantly influenced subsequent meta-learning techniques, which continue to play a pivotal role enabling efficient and adaptable learning across diverse tasks.
tasks with minimal fine-tuning, MAML employs
Hospedales
2021
3 Methods
3.1
Transfer Learning in PINNs
Transfer learning has emerged as a valuable technique to enhance the efficiency and scalability of PINNs. By leveraging knowledge from pre-trained models, transfer learning addresses the compulational challenges and convergence issues often encountered when training PINNs from scratch. This section reviews key advances in the application of transfer learning to PINNs, with an overview of the literature discussed provided in Table M
3.1.1 Full Fine-tuning
Full model fine-tuning (FFT) a transfer learning strategy that adapts a pre-trained model to a new task by updating all its parameters. FFT facilitates efficient adaptation of PINNs, particularly when the source and target tasks are similar. This approach is especially useful in computationally intensive scenarios where training PINN from scratch for each new task would be prohibitively expensive.
8

---

Published in Transactions on Machine Learning Research (03/2025)
An example of full model fine-tuning in PINNs is the TL-PINN method introduced by Prantikos et al. (2023), which addresses the Point Kinetic Equation critical tool for real-time reactor analysis. TL.- PINN employs transfer learning to accelerate training by pre-training PINN on a source task, such as simulating reactor behavior under nominal conditions or controlled parameter variations (c.g., temperature shifts). The pre-trained model is then fine-tuned for a larget lask involving distinet transients, such as different reactivity insertion schedules. TL-PINN has been shown to reduce training iterations by an order of magnitude compared to conventional PINNs. Performance improvements correlate with task similarity, measured using geometric metrics like the Hausdorff distance. This framework provides a practical guideline: TL-PPINN significantly improves efficiency without sacrificing accuracy when source and target tasks share dynamical features (e.g,, temporal profiles of neutron flux).
Zhou Mei 2023 proposed a method that combines the smoothed finite element method (S-FEM) with PINNs address inverse problems, specifically the inversion of material parameters in scarce data regimes. In such regimes, a common strategy is to pre-train on solver data and fine-tune with limitod, partial observations. The authors use S-FEM to generale high-quality data for pre-training the PINN, and then, as proof of concept, they fine-tune the model with additional S-FEM data to infer new parameters and evaluate their approach. S-FEM is preferred over FEM as it generates higher-quality data, which, as demonstrated in their results, enhances the pre-trained model and ultimately improves fine-tuning accuracy. Their findings indicate that the use of transfer learning with PINNs increases computational efficiency by a factor of two compared to standard PINNs. The authors emphasize the importance addressing negative transfer in transfer learning, noting that the success of transfer learning depends on the assumption that the source and target domains share similarities.
Table Transfer Learning in Physics-informed Neural Networks.
Transfer Learning Strategics in PINNs
FS.
Literature
Task
PT
PType
Benchmark Equations
FFT
Prantikos et al. 2023
ODE
ST
Fwd.
*Point Kinetic (PKEs) *Elastoplastic
Zhou & Mei
2023
PDE
ST
Inv.
PEFT
Desai et al. 2021
PDE/ODE
MT
Fwd.
Pois. 2D Schr. 1D 1st/2nd-order ODEs
Goswani al. 2020
PDE
ST
Fwd.
*Fracture Mechanics2D
Gao
(202
PDE
ST
Fwd.
Linear Parabolic Allen Cahn
Pellegrin et al.
(2022
ODE
MT Fwd.
Stochastic Branched Flow"D
Chakraborty (2021
PDE/ODE
ST Fwd.
Stochastic ODE, Burgers" ID
CTL.
Lin & Chen (2024)
PDE
ST
Inv.
Schrödinger
Xu et al. (2022)
PDE
MT
Inv.
2Elastic-33D Hyperelastic
Mustajab el al.
(2024)
ODE/PDE
ST Fwd.
Harmonic Oscillator, Wave
Equation" 1D
Note: Fine-tune Strategy (FS), Pre-train type (PT), Problem Type (PType), Full Fine-tune (FFT);
Parameter-cffiiient Fine-tuning (PEFT), Curriculum Transfer Learning (CTL), Singlo-task Learning (ST), Multi-task Learning (MT). Equations with (+) are domain-specifie problems. References marked with (t) indicate the use of few-shot learning techniques. Abbreviations: Poisson Pois., Schrödinger Schr.
3.1.2 Parameter-Efficien Fine-tuning
PEFT techniques were originally introduced as a mothod to fine-tune largo-scale pre-trained models while minimizing computational and memory requirements. These methods typically introduce a small set of trainable parameters, such as adapters low-rank updates, allowing for efficient adaptation. In this context, the term PEFT is adapted to describe selective fine-tuning strategies within PINNs that aim to reduce computational costs by limiting the number of updated parameters. Although the scale of PINNs may not
4The Point Kinetic Equation simplified model used analyse the behavior of nuclear reactors over time (reactor transients). consists of system of stiff nonlinear ordinary differential equations that model the kinetics reactor variahles.
9

---

Published in Transactions on Machine Learning Research (03/2025)
align with the large models typically associated with PEFT, the underlying principles of parameter efficiency and adaptability remain relevant. The studies discussed here illustrate how these principles can enhance the scalability and performance of PINNs, particularly in data-scarce or resource-constrainge scenarios.
In the context of PINNs, Desai et al. 2021) proposes a pre-training and fine-tuning strategy for PINNs to efficiently solve linear ODEs and linear PDEs. During the pre-training phase, the method learns a set of shared bases, represented as ((x), which is derived from the hidden layers (H1-1 11=-2 o...o o (x), by training on multiple source tasks involving ODEs or PDEs. During the fine-tuning phase, the focus shifts to determining the expansion coefficients, represented by the weights of the output layer new problem instance. If a can be obtained analytically in closed form, it is computed by solving a linear system of equations, reducing the computational cost. to a matrix inversion. For more complex cases, a is optimized through gradient descent, where only the output layer a is updated, while the shared bases Φ(x) remain frozen. The final solution ū(x) is expressod as:
Wout, for a
(a( = a- ((x).
(8)
This approach enables efficient transfer of shared bases ((x) between tasks and significantly reduces training overhcad. The accuracy of the final solution depends on how well the bascs ((x) cover the solution space of the target problem. By calculating the coefficients in closed form for linear systems, this method achieves one-step adaptation, allowing the shared bases ((x) to be adapted to new problems without iterative training or updates to the hidden layers. Future directions of this work include extending the method to incorporate real-world observational data, expanding the framework to nonlinear PDEs, and exploring the characteristics of shared bases to provide better generalization and adaptivity across tasks.
Goswami et al. (2020) proposes a method for phase-field fracture modeling using PINNs enhanced with transfer learning. Unlike conventional residual-based PINNs, this approach minimizes the variational energy of the system while enforcing boundary conditions as hard constraints. This formulation offers two key advantages: 1) imposing boundary conditions is simpler and more robust, and 2) the resulting equations involve lower-order derivatives, making training faster. The fracture modeling process involves iteratively updating the displacement field u, which describes material deformation, and the phase field d by minimizing the total energy at each small displacement step. To overcome the high computational cost of retraining the PINN at every step, transfer learning is employed: only the last layer's weights and biases are retrained, using the previous step's parameters as initialization. This gransfer-learingg approach significantly improves computational efficiency compared to standard PINNs by requiring fewer iterations achieve convergence and substantially reducing the time required for cach iteration.
Chakraborty (2021) proposes transfer learning approach to approximate high-fidelity models using PINNs. The method begins by training a PINN on a low-fidelity model of a given IBVP task. Then, a transfer learning technique is applied, where only the last one or two layers of the network remain trainable. This pre-trained model is subsequently fine-tuned to approximale a higher-fidelity model using limited high- fidelity observations of the same task. The approach is particularly useful in scenarios where the exact high-fidelity model is not known a priori, allowing efficient adaptation different boundary conditions or initial conditions.
Gao et al. 2022) introduces SVD-PINNs, transfer learning method designed to solve high-dimensional PDEs efficiently. The core idea involves pre-training a PINN, which consists of a three-layer MLP, on an arbitrary PDE task. During fine-tuning for a new PDE, the weight matrix of the middle layer is decomposed using SVD (Singular Value Decomposition). The singular vectors, which capture intrinsic patterns from the source task, are kept frozen, while the singular values and weights of the initial and final layers are adapted. The authors demonstrated the effectiveness of SVD-PINNs in 10-dimensional linear parabolic equations and Allen-Cahn equations, showing superior performance in terms of relative error and convergence speed compared to vanilla PINNs and other transfer lcarning methods. A notable advantage of SVD-PINNs is their elficiency in solving multiple related PDEs with identical differential operators but different right-hand side functions. However, the main challenge lies in optimizing the singular values during training. Successful
6Phsso-ficld modcling tracks fracture using a continuous scalar field o. This scalar field represents the damage state of the material, smoothly transitioning from intact fractured material, and models the evolution of cracks without sharp discontimities.
10

---

Published in Transactions on Machine Learning Research (03/2025)
optimization of these values leads to better performance than methods that freeze the first layer, as seen in previous transfer learning approaches for PINNs. In contrast, biased or inaccurate singular values can result in worse outcomes than prior methods.
Pellegrin et al. 2022) presents a multi-task learning strategy aimed at improving training efficiency and performance by leveraging knowledge from related tasks. Their approach employs a multi-head architecture with two primary phases: pre-training and fine-tuning, Initially, during the pre-training phase, the multi- head model, comprising shared base neural network along with multiple task-specific heads, is trained concurrently various related PDE tasks. Through training on these interrelated tasks concurrently the shared base network learns to extract relevant features and representations that encapsulate the underlying dynamics common across the lasks. In the subsequent fine-tuning phase, the weights of the pre-trained base network are fixed, and a new task-specific head is introduced and fine-tuned for the target transfer learning task. This fine-tuning phase enables the model to adjust the learned shared representations to the specific requirements of the new task while leveraging the knowledge gained from related tasks during pre-training. By leveraging the shared base nelwork learned from multiple related lasks, the model can potentially converge faster and achieve better performance the target task than training a PINN from scratch.
3.1.3 Curriculum Transfer Learning
Curriculum Transfer Learning involves progressively increasing the complexity of tasks during the transfer process, starting with simpler tasks and moving towards more complex ones. An example of this approach is the work of Mustajab et al.] (2024). who employed a curriculum transfer learning strategy to address high-frequency and multi-scale problems. The method begins by training on relatively simple low-frequency problems, which are easier to solve, and then gradually escalates to more challenging high-frequency tasks, transferring knowledge gained from the lower-frequency problems. Through full-weight transfer learning, this strategy successfully enabled the model to learn high-frequency ions that traditional PINNs cannot achieve without increasing the number of layers. However, the authors highlight the importance of under- standing the limitations of transfer learning, particularly in terms of when succeeds and when may fail. Overall, this approach enhances the convergence speed and robustness of PINNs for high- feghfrreuency and multi-scale PDEs.
Gradient-enhanced PINNs (gPINNs), introduced by Yu et al. 20222 improve standard PINNs by incorpo- rating the gradient of the residual as an auxiliary loss term. This approach captures local information around collocation points, enhancing accuracy but increasing computational overhead due to added optimization complexity. To address this limitation, Lin & Chen (2024) proposes TL-gPINNs, an extension of gPINNs tailored for inverse problems with variable coefficients, such as time-dependent boundary conditions. TL- gPLNNs adopt a two-step optimization strategy: they first train a standard PINN on a simplified objective and then fine-tune the whole model using the gPINN loss. This approach reduces both error and com- putational cost compared to training gPINNs from scratch. In particular, experiments demonstrated that for PDEs involving multiple loss terms, standard PINNs can sometimes outperform gPINNs, as the addi- tional loss terms increase the optimization complexity. By progressively introducing complexity through the gPINN objective, TL-gPINNs employ a curriculum learning framework to balance computational efficiency and solution accuracy.
Xu et al. 2022) addressed the challenge of inverse analysis in engineering structures, where acquiring data for structural components is often expensive. To this end, the authors sought to improve the training efficiency and accuracy of PINNs for inverse problems through a multi-task transfer learning approach. Their proposed solution involves two-stage lcarning process. Initially, in the pre-training stage, a simplified loading scenario is used to pre-train the PINN model, including both the model weights and task-independent loss balancing weights. Subsequently, in the fine-tuning stage, the pre-trained model is partially fine-tuned with data from real engineering problems, with only the last two layers updated. The method was applied to 2D lincar and hyperelasticity problems in solid mechanics. Combining layer freezing with inherited multi-task weights from pre-trained models significantly accelerated training convergence.
11

---

Published in Transactions on Machine Learning Research (03/2025)
3.2 Meta-learning in PINNs
The integration of meta-learning techniques with physics-informe neural networks has shown promise in improving model adaptivity and generalization. Table 2 presents the taxonomy used to study these tech- niques in the context of PINNs, focusing on the meta-learning representation ( ('what is being meta- veetalearnnd") (Hospedales et al. 2021 In the following section, various studies are introduced according to this taxonomy.
Table 2: Meta-learning Strategics in Physics-informed Neural Networks.
Meta-lcarning Strategies in PINNs
Type Approach
Literature
PType
Equation
Liu et al. 2022
Both
Pois. 1-2D.1-2D, [Burg., Schr.J
Zhong al. [2023
Fwd.
*Plasma Sim. 1D
Weight FFT
Penwarden et al.
(2023)
Fwd.
[Burg., Hcat] [A-C, D-R]
Init.
CCeng & Alkhalitah
20024
Fwd.
Wavefield
Qin et al.
2022
Fwd.
Burg. ID [Pois., Hyp.-elast.
PEFT
Cho et al.
2024b)
Fwd.
C-D-R'D, Helm. 2D
Lay./Nur:
Chen et al.
2021
Inv.
A-D-R
Net.
Bischol & Kraus
2022
Fwd.
Pois. 2D [Burg. K-G, A-CJ
Activations
Struct.
Chen & Kooby
2024
Fwd.
Sampling
Toloubidokhti al.
2023
Fwd.
[Bugg., Conv., R-D] Helm. 2D 2- Nonlinear PDF10D Burg. ID [Max., Laplace.
Points/Params
Tang et. al.
[2023
Fwd.
Input
Huang et al.
(0222)
Fwd.
Latent Rep.
Twata et al.
2023
Fwd.
Arbitrary Param. PDE1
Param. Loss
Psaros et al.
2022
Both
[Adv., Burg. ID SS R-D2D Burg. 1D, [LDC Flow, Pois.J2D
Loss
Loss Attention
Song
2024
Fwd.
Note: Problem Type (PType), Forward Problem (Fwd.), Inverse Problem (Inv). Equations marked with
(*) represent domain-specific problems. References marked with (f) indicate the use of few-shot learning techniques. Abbreviations: Poisson Pois., Burgers Burg., Schrödinger Schr., Simulation Sim., Allen-Cahn = A-C, Diffusion-Reaction D-R, Drrvection-Difusionn-Beeee Adveetion-DDiftusion--eee
= C-D-R, Helmholtz -Helm., = A-D-R, Klein-Gordon = K-G, Diffuusson = R-D, Elliptic = Ellip., Hyper-elasticity = Hyp.-elast., Maxwell = Max., Parametric = Param., Advection = Adv., Steady State =
SS, Lid-driven Cavity L.C.
3.2.1 Learning the Weight Initialization
Starting from good weight initialization can lead to faster convergence, better accuracy, and reduced computational costs a key factor for real-time applications. A well-informed initialization not only reduces training time but also improves generalization, making it a central focus in many studies. Meta-learning for weight initialization, building on transfer learning principles, extends this by learning from the training process across multiple tasks. Techniques such as MAML, Reptile, and hypernetworks identify or generate optimal starting weights, enhancing efficiency and adaptivity. Using these approaches, PINNs can achieve scalable, resource-efficiett performance across diverse problem domains. This section reviews these strategies and their impact on advancing the effectiveness of PINNs.
Liu et al. (2022) employs the Reptile algorithm Nichol et al. 2018) to find good initialization of the param- eters and compares it against other initialization techniques, such as Xavier initialization, in unsupervised, supervised, and semi-supervisod settings for the forward and inverse problems. In their experiments, their Reptile weight initialization outperformed Xavier initialization, with the unsupervised Reptile approach per- forming the best. The authors point out that this initialization be used with other PINN architectures
12

---

Published in Transactions on Machine Learning Research (03/2025)
and that, in addition to finding a good initialization of the network parameters, this method could serve to provide good starting points for works that use adaptive activation functions or losses.
Zhong et al. 2023) and Cheng& & Alkhalifah (024) both adapted the MAML approach by Finn et al. (017)) to PINNs, aiming to reduce the training steps required for new tasks by leveraging a meta-network. for weight
initialization. However, their implementations and benchmark problems differ slightly. Both approaches
utilize a support set in the inner loop to update the network for a specific task and query set in the outer loop to optimize the weight initialization. The key distinction lies in how these sets are constructed: Zhong et al. (2023) uses the same PDE task for both the support and query sets, evaluated at different collocation points, while Cheng & Alkhalifah (2024) uses different PDE tasks for the two sets. In terms of benchmark problems, [hhong et al. (2023) applied their approach to plasma simulations, demonstrating that generalization performance strongly depends on the similarity between source and target tasks. In contrast,
Cheng & Alkhalifah (2024) focused on seismic wave equations, showcasing faster convergence and higher prediction accuracy compared to vanilla PINNs. However, both methods rely on an initial meta-training phase to establish robust network parameters. This phase involves two gradient computations, one for the inner task and one for meta-initialization, making the process memory-intensive and computationally costly, Cheng & Alkhalifah (2024) further emphasizes the importance of task diversity during meta-lcarning finding that incorporating more tasks helped capture a broader range of distribution features, leading to improved generalization. They also observed that using 20 iterations in the inner loop yielded better performance compared to fewer iterations. Finally, they highlighted the potential for combining their approach with other PINN architectures to further enhance adaptability and efficiency.
Qin et al. (022) compared MAML and LEAP (Flennerhag et al., 2018) weight initialization with PINNs, extending tasks to different geometries and boundary conditions. LEAP, similar to MAML and Reptile, is a general meta-learning framework. However, instead of finding a good initialization based solely on the final state of the weights from different tasks, LEAP considers the entire optimization path. The objective is to minimize the expected length of the path traveled during the training process, allowing more efficient knowledge transfer between learning processes. In their work, they found that MAML outperforms LEAP in accuracy for a given runtime and requires less hyperparameter tuning. However, LEAP's meta-training is faster and less memory intensive.
Penwarden et al. 2023 compared different weight initialization stralegies to improve the optimization of PINNs in terms of both time and accuracy. The methods included random initialization, MAML, center initialization, and initialization via interpolation of pre-trained PINN weights. Center initialization involves starting with the pre-trained weights at the center of the parameter space, assuming that this central position is a reasonable initial point for optimization. For the interpolation method, multiple pre-trained PINN weights were interpolated to infer general weight initialization that would lead to a more accurate and faster convergence. The conclusions of this work were that the interpolation methods provided good initial weights, enhancing the optimization performance in terms of accuracy and time. However, definitive conclusion could be drawn regarding the superior interpolation method, as performance varied across different PDEs between Spline, RBF (cubic, Gaussian, and multiquadratic), and polynomial interpolation. One surprising finding was that initializing the weights with those of a pre-trained PINN, trained at the center of the PDEs
parameter space, produced results comparable to the interpolation methods for and, for the 1D-task, outperformed MAML. The authors note that their approach assumes that the parameter space is well-behaved, meaning it does not change drastically between parameters, and emphasize that for future work, identifying the boundaries of these regions is important.
Cho et al. 2024bb developed the Hyper-Low-Rande Rank PINN, which combines leta-learning with PEFT to address parametric PDEs more efficiently. The method features a two-phase training process. In the pre- training phase, the weights of the hidden layers of the base model are constructed using an SVD approach, W UV,, where Σ (the singular values) are provided by the meta-network and the singular vectors and V are part of the base model. The first and last layers of the base model are kept as standard linear layers. During the fine-tuning phase, the meta-network generates adaptive weights for new tasks and trims less significant weights to maintain a compact, hyper-low-rank structure. In addition, the first and last layers are optimized along with the adaptive weights. The authors conclude that their method improves efficiency and accuracy by leveraging meta- neta-learning and low-rank approximations, reducing computational and memory
13

---

Published in Transactions on Machine Learning Research (03/2025)
costs. It effectively handles varying PDE parameters, mitigates failure modes, and outperforms standard PINNs. According to the authors, future work should focus on extending the framework to handle more general settings, such as parameters that define initial / boundary conditions or different domain shapes, and improving the expressiveness of the bases for greater adaptability across a broader range of PDEs.
3.2.2 Learning the Network Structure
Learning the network structure involves tailoring the architecture of neural networks, such as the number of layers, hidden dimensions, and activation functions, to specific tasks or domains of problems. In the context of PINNs, meta-learning strategies have been employed to dynamically adapt the network structure to the unique requirements of different PDE tasks, enhancing the convergence and performance of PINNs, as demonstrated in the following works.
Chen et al. [2021) developed a method to solve the mixed problem (forward/inverse) of the advection- diffusion-reaction (ADR) system using sparse measurements. Given the stochastic nature of the problem, they utilized a PINN architecture called sPINN (Zhang et al. 2019). This composite network comprises several uub-net each corresponding to different fluid flow frequencies (modes). Due to the complexity of determining the optimal number of layers and neurons for each sub-network, Neta-Bayesian optimization was employed to automate the selection process. The study focuses on realistic scenarios with limited or sparse data, addressing both forward and inverse problems common in engineering, where some material properties and sensor measurements are known. According to the authors, future work should address uncertainty quantification and explore other optimization methods, such as genetic algorithms, greedy methods, or reinforcement learning, to improve NN architecture.
Bischof & Kraus (0022) proposed method that combines Nixture-of-Experts (MoE) and PINNs to solve a single task. In this approach, multiple expert PINNs are trained on different partitions of the input space, and a gating network learns the optimal weighting of their predictions. This allows experts to specialize in different regions of the imput space, potentially improving overall accuracy. The meta-learning aspect lies in modulating the gating mochanism to balance the contribution of cach expert PINN, thereby enhancing the accuracy and convergence of the overall model. Some key remarks of this work are that, when different experts have different architectures, the gating mechanism discarded the networks with tanh activation and favored sine activation. Another takeaway is that regularization that weights the importance of each expert is crucial; in this case, the optimal solution was achieved with three learners. With this method, the focus was on improving convergence and accuracy. This study indicates that future research should be concerned with increasing performance, efficiency, robustness, and scalability.
Chen & Koohy (2024) introduced GPT-PINN, which combines meta-learning with a task sampling strategy that dynamically expands shared basis dictionary. This method aims to solve new parameter instances u(æ, t;;1 by approximating them a weighted sun of pre-trained PINNs:
u(x, ti ~ adlH)
Here, 6% represents pre-trained PINNs at different parameter configuratioss, and as(μ) are parameter- dependent coefficients learned by a meta-network. The meta-network modulates the influence of each pre- trained PINN basis 6% through the coefficients (H) for a given PDE instance. If the approximation fails to meet the accuracy criterion, a new full PINN is trained individually for that parameter instance, and its solution is added to the set of basis functions &&N, thus continuously expanding the generalization range of the overall structure. The approach enables efficient adaptation to new parameter instances while contimu- ously improving the model's generalization capabilities by expanding the shared basis dictionary, ultimately reducing the computational cost and enhancing the performance of PINNs across diverse parameter spaces.
3.2.3 Learning the Loss Function
Meta-learning techniques have also been used to optimize loss functions for PINNs, offering an approach significantly improve their performance. These strategies go beyond traditional fixed loss formulations by
14

---

Published in Transactions on Machine Learning Research (03/2025)
adaptively learning loss functions tailored to the specific needs of diverse PDE tasks. By dynamically adjust- ing the weighting of errors, meta-learning enables better handling of challenging problem regions, improving convergence and enhancing the generalization capabilities of PINNs. The following works demonstrate how these meta-lcarning strategics have been successfully applied to discover effective loss functions and improve PINN performance across a variely of problem domains.
Psaros et al. (0222) proposed a gradient- baaee meta-learning algorithm that discovers effective loss functions for various PDE problems. The algorithm operates in two phases: meta-training and meta-testing. During the meta- meta-training phase, parametric loss function is optimized over a distribution of PDE tasks, learning the optimal parameter values that generalize well across the task distribution. In the subsequent meta-testing phase, the meta-learned loss function is employed to train PINNs on unseen PDE tasks that may differ from the original task distribution and PINN architectures used during meta-training. This meta-learning approach significantly improves the generalization performance of PINNs, enabling them to achieve high accuracy even on -ustr--diittibution scenarios and PINN architectures that were not encountered during mela-training.
Another recent work that uses meta-learning for the loss function is the Atsennttttt PINN (LA-PINN) by Song et al. (2024). LA-PINN treats the loss function as a learnable component employing multiple loss- attentional net works (LANs) that are adversarially trained alongside the main PINN model. While the main network minimizes the loss via gradient descent, the LANs use gradient ascent to meta-learn point-wise weights for the loss terms, effectively discovering an 'attentional function" to distribute different weights to each collocation point error. This loss-attentional meta-learning framework allows tailoring the loss function per problem by leveraging experience from related tasks, potentially enhancing PINN performance over fixed hand-crafted losses. The adversarial training process, inspired by Generative Adversarial Networks (GANs), enables the LANs to assign higher weights to stiff or fard-to-fit regions, aiding convergence by emphasizing the challenging areas during optimization.
3.2.4 Learning the input
Learning the input involves adapting the input data provided to neural networks. In the context of Physics- Informed Neural Networks, this can refer to selecting and adjusting the number and locations of collocation points. Additionally, during a two-phase training process, another strategy is to adapt the task sampling strategy in the pre-training phase. A third approach involves self-referentall learning, where the meta-learner dynamically adapts parts of the input based on the specific task requirements.
Toloubidokhti et al. 2023) indicates that many existing meta-learning strategies neglect the varying difficulty levels across different tasks and propose that depending on the difficulty, different collocation point positions and densitics should be employed accordingly. To address this, they developed a Difficulty-Awar Sampler (DATS) for meta-learning of PINNs, which aims to optimize the lask sampling probabilities during meta-training to minimize the average performance across all tasks during meta-validation DATS employs two strategics: adaptively weighting PINN tasks based on their difficulty, allowing more challenging tasks to contribute more to the meta-learning process, and dynamically allocating the optimal number of residual points (collocation points) across tasks, ensuring that more difficult tasks receive a higher collocation point budget. The evaluation of DATS against uniform and self-paced task-sampling baselines on two meta-PINN models across four benchmark PDEs demonstrates that it improves the accuracy of meta-learned PINN solutions and reduces performance disparity among different tasks while using only fraction of the residual sampling budget required by the bascline methods.
Task
Another significant contribution is made by Tang et al. 2023 who proposed Deep Adaptive Sampling (DAS) method for solving high-dimensional PDEs using PINNs. The key innovation is treating the residual of the PINN as probability density function, approximated by deep generative model called KRnet. The DAS method involves two main components: solving the PDE by minimizing the residual loss on the current set of collocation points and adaptive sampling, where new collocation points are sampled from the KRnet distribution. This method places more points in regions with high residuals, similar to classical adaptive methods like adaptive finite elements. By iteratively refining the training set, the PINN retrains an updated set with focus on high-error regions. The KRnet model is meta-learned to approximate
15

---

Published in Transactions on Machine Learning Research (03/2025)
the residual distribution, enabling adaptive sampling tailored to the current PINN solution. The approach is particularly effective for low regularity and high-dimensional PDFs, where uniform sampling is inefficient. The authors provide a theoretical analysis showing the DAS method can reduce error bounds, with numerical experiments demonstrating significant accuracy improvements compared to uniform sampling.
Self-referential meta-learning approaches have also been combined with PINNs. The Aata--Aut--ecooba (MAD) introduced by Huang et al. (2022) utilizes a self-referential approach to learn parametric PDE.s. This approach involves pre-training neural network uz z) to approximate parametric PDE solutions using a physics-informer loss, where x is tunable input latent vector that implicitly encodes PDE parameters p. After pre-training, the network is fine-tuned for new PDEs by either fixing weights 0 and tuning z or tuning both z and 0, allowing it to search for solutions on or near the learned solution manifold.
Instead of learning the latent representation implicitly within the same PINN network, [wata et al. (2023) proposed to leverage multiple meta-networks to encode the governing equations and boundary conditions into a latent vector z. The fine-luning strategy involves initializing the latent vector z with the help of the meta- networks while keeping these meta-networks frozen. The initialized latent vector is then set to be tunable, and both the PINN weights O, and the latent vector z are fine-tuned for the new PDE task. In both MAD and Iwata et al. (002)), the PINN weights are initialized according to the final pre-training phase.
3.3 Few-shot Learning in PINNs
Few-shol learning is a machine learning paradigm that enables models to generalize effectively from very limited training examples, emphasizing data efficiency and adaptivity to new tasks or domains with minimal supervision. In the context of PINN adaptivity, this involves addressing three key aspocts: the number of pre-training tasks, the number of collocation points required for training, and, when regression data is used, its reduction- the data may include available observations, sensor measurements, or ground truth data. By strategically selecting and sampling tasks, the model can achieve optimal performance during fine-tuning with fewer samples, thereby enhancing its ability to generalize across tasks. Another approach focuses on optimizing the spatial or temporal distribution of collocation points, reducing computationa ional costs while maintaining reliable predictions in under-sampled regions. Additionally, in semi-supervised or supervised settings, methods leverage sparse observations (e.g., sensor data) to reconstruct full solution fields or solve inverse problems, making them particularly suitable for real-world applications where data is inherently scarce. This section highlights works that integrate meta-learning and transfer learning (marked with the 'f' symbol in Tables and 2 to achieve few-shot lcarning, prioritizing adaptable, generalizable, and data-efficient methods for solving diverse PDE problems.
For example, Chen & Koohy 2024 focuses on reducing pre-training samples by gradually selecting tasks with the highest residuals, thus optimizing task sampling for improved performance during fino-tuning. Similarly, Cheng& & Alkhalifah (2024) investigates the impact of pre-training sample sizes, finding that increasing the number of samples improves performance, likely due to the greater diversity of features captured in larger datasets.
On the other hand, Toloubidokhti et al. (2023) develops a technique that not only optimizes task sampling based on task difficulty but also dynamically allocates collocation points according to the complexity of the task. This dual approach improves the efficiency of meta-learning for PINNs by addressing task sampling and collocation point allocation.
Finally, high-frequenc, problems often require a denser distribution of collocation points. In the work of Mus- lajab al. (2024), the authors demonstrate that training with curriculum starting from a low-frequency problem and gradually progressing to high-frequency learn the high-frequency problem without incrcasing the number of collocation points. This highlights how sicicuulum-basedd approaches can reduce the need for high-density collocation points, benefiting few-shot learning scenarios.
-niillem-eabbesseeeeeeeeeeeeeeee the final model to successfully
Building on the goal of reducing data requirements in real-world applications, several studies have developed strategies to use sparse observations effectively. These methods aim to reconstruct solutions or tackle inverse problems with limited supervision. The following studies are particularly relevant in this context:
16

---

Published in Transactions on Machine Learning Research (03/2025)
Zhou & Mci 2023 explored whether PINN could be pre-trained using small dataset generated by solver and then fine-tuned with limited observational data to solve inverse problems, which is key aspect of few-shot learning. They used solver-generated data during the fine-tuming phase rather than relying on real-world observations. Their work highlights the potential of transfer learning in data-scarce scenarios, offering valuable insights for applications with limited data availability and minimal supervision.
Chakraborty (202I) frames the approach around the assumption that field data is scarce in practice. To address this, the work blends concepts from PINNs and data-driven learning. The primary idea is to first train low-fidelity model based on the PINN loss and then apply transfer learning to update the model using high-fidelity data. With this approach, the pre-trained model is trained without data, while fine-tuning is performed with only a few data samples.
Similarly, Xu et al. (2022) addressed the challenge of limited data availability by employing multitask pre- training on simplified tasks solvable with numerical solvers, effectively implementing a few-shot lcarning approach. This stralegy enabled the model to leverage abundant and low-cost data for the fine-tuning phase, thereby reducing the number of samples needed for real-world inverse problems with sparse observational data. The study underscores the potential of transfer learning to significantly minimize sample requirements, core principle of few-shot learning.
Though some of these studies did not explicitly benchmark few-shot learning, they collectively demonstrate how transfer learning and meta-learning can address limited-data problems, ultimately reducing the number of samples required for model training and fine-tuning. This highlights the potential of these approaches enable efficient learning in scenarios with scarce data, central tenet of few-shot learning.
4 Metrics & Benchmarks
This section introduces the benehmarks and metrics relevant to efficient model adaptation. Section presents the common benchmark PDE problems used throughout the works herby surveyed, Section 4.2 outlines error quantification methods for single-task and multi-task scenarios, and Section 4.3 metrics for assessing efficient adaptivity, crucial for evaluating model performance.
presents key
4.1 Benchmark PDE Problems
Several methods evaluate their performance using a variety of benchmark problems to ensure robustness and reliability. Tables I and 2 compile the specific equations adopted in individual studies, providing a compre- hensive overview of the problem domains explored. Building this, Table ! (available in Appendix AA.1) synthesizes the broader landscape of benchmark usago, categorizing equations by their complexity, key drivers of difficulty and prevalence in the literature. This table serves as a practical guide for selecting appropriate benchmarks based on specific requirements for complexity and solution characteristics.
The most commonly used equations in this survey are Burgers' equation (featured in studics) and the Poisson equation (5 studies), which represent intermediate and baseline levels of complexity, respectively. Less frequently used benchmarks include the Allen-Cahn and Schrödinger equations (each used in 3 studies), while the A-D-R and Navier-Stokes equations, due to their higher complexity, are each utilized only once. The popularity of Burgers' equation can be attributed to its balanced combination of manageable com- putational demands and challenging nonlinear dynamics, making it particularly well-suited for evaluating time-dependent models.
4.2 Error Quantification in PINNs
The quantification of errors in PINNs has traditionally relied on the relative L2 error, which measures deviations from ground-truth solutions but does not assess generalization across tasks. While effective in single-task scenarios, multi-task settings require a focus on task similarity- a key factor in evaluating interpolation and extrapolation within the parametric space, as well as generalization through knowledge transfer. To address this, Chen al. (2021) presented the errors in Table 3 incorporating not only single- task errors and losses but also the 'worst-case' error, which captures the largest error within set of tasks.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Additionally, they visualize results cross-task distributions, such as candlestick plots, providing a more comprehensive view of error variation. Optimizing with respect to the 'worst-case' loss further aids training by reducing worst-case errors and enhancing generalization in multi-task scenarios.
A central challenge in benchmarking task similarity lies in rigorously quantifying it. Current methods exhibit key trade-offs. For instance, [Huang al. (2022) pre-train models on PDE parameters drawn from divergent distributions, but this heuristic does not guarantee task dissimilarity, as differing PDE parameters may still result in similar solutions. In contrast, Prantikos et al. (2023) uses geometric metrics such as the Hausdorff distance to measure the dissimilarity between solutions across tasks. This approach involves comparing the solutions of set of tasks to identify OOD tasks, which are primarily used for benchmarking purposes. However, this method requires finalized solutions, limiting its applicability to scenarios where solutions are already available. Alternatively, adaptive frameworks, such as GPT-PINN (Chen & Koohy 2024), bypass explicit similarity metrics by identifying OOD tasks based on the initial task-specific loss and retraining models when this loss exceeds prodefined thresholds, prioritizing practicality over formal guarantees.
These insights emphasize the importance of integrating task similarity, error distributions, and extreme-valus analysis into error assessment frameworks. Such integration is essential for robust model development and for gaining deeper understanding of generalization, particularly in multi-task settings where knowledge transfer and pateroolation/etrrppllllt within the parametric space are key considerations.
Table 3: Evaluation Metrics for Multi-Task PINNs. "Worst-case" metrics compute the maximum over lasks (parameterized by μ)) while "Task-wise" metrics are evaluated per task at the final iteration.
Metric
Description
Equation
Worst-case Loss
Maximum loss across tasks
max ((
nET
Terminal Loss
Task-specific loss at final iteration
C(us: p1)
Uno (n)
Ugt (4)l12
Worst-case Rel. L2 Error
Maximum relative error across tasks
max WET [1o(
Ugge (1) 12
Ugt (14) 1
Terminal Rel. Lg Error
Tpsk-specific relative error at final iteration
Ugg (p) ll2
Terminal Abs. Error
Task-specific absolute error at final iteration
(o)
Ugt (14)11a
Deeson By Vanila PINN
Midel Adaapaaion Ousgasioon
Losee: PINN leaming
Arg Timee
vontllaa plnn
Avy. Epodia
monta leccning pita
10
Avg. Loon
68-
Model Parameieess
Pennzz Eq: Meta- Leastan FINN
20.1
RL2 Eroe
Callocation
200
300
Mpocce
vanilla
MACs
mert--larringg pinn
Figure 4: Example of efficient model adaptation through meta-learning. The blue-circled metrics represent key factors that efficient adaptivity aims to reduce, influenced by the other metrics in the radar chart. On the far right is the predicted solution after only 100 epochs.
18

---

Published in Transactions on Machine Learning Research (03/2025)
4.3 Efficient Adaptivity Metrics
Evaluating the adaptivity and efficiency of PINNs is crucial for practical applications. Key metrics assess data requirements and computational efficiency, focusing on minimal data usage and reduced training time. Figure f compares a vanilla PINN, trained from scratch for each task, with a meta-learning PINN that employs a hypernetwork- based adaptation strategy to leverage prior knowledge (Cho et al. 2024b). The radar chart provides a normalized comparison of key factors influencing efficient adaptation, including the number of epochs, model parameters, collocation points, model complexity, and end metrics such as average convergence time, final loss, and final error. The reported values were obtained by training the models until either target loss of 0.05 or maximum of 1,200 epochs was reached. The results demonstrate that meta- learning significantly reduces training time, epochs, and collocation points, thereby lowering computational overhead.
4.3.1 Data Efficiency
Collocation Point Budget. The number of collocation points significantly impacts the convergence speed and accuracy of PINNs. Some studies use a fixed number of collocation points, while others employ adaptive sampling strategies, allocating more points to regions with higher PDE losses (Toloubidokhti et al. 2023). A useful evaluation strategy is to measure the accuracy of different sampling techniques given the varying budgets of the collocation points (Wu et al., 2023 By examining the trade-off between the number of collocation points and the resulting accuracy, it is possible to optimize the sampling strategy to suit specific problems and computational constraints, enhancing the training performance and accuracy of PINNs.
Observation Point Budget In many applications, only a limited number of observation points are available for analysis. Rather than relying solely on unsupervised loss, it is important to make the most of these scarce observations. To address this, a metric is designed to assess the relationship between the number of observation points and the accuracy of the analysis. Specifically, it evaluates the accuracy that can be achieved given fixed budget of N observation points. This is particularly crucial for real-world scenarios where dala availability is constrained.
Task Sampling Strategy. Several PINN techniques employ a two-phase training process consisting of a pre-training phase and a fine-tuning phase. The pre-training phase can be conducted using cither a single instance or multiple instances of PDEs. To achieve better generalization across a distribution of tasks, it is crucial to pre-train on multiple tasks distributed along a given parameter range. However, determining the optimal sampling strategy for selecting pre-training tasks is a challenging task. The goal is to identify the fewest number of pre-training tasks that yield the best results, the number of pre-training tasks influences both the pre-training resources and the final fine-tuning accuracy. Recent works, such as that of Toooubidokhtt et al. (2023), have attempted to address this problem. One approach is to mcasure the 'per performance disparity" within a given range of lasks, defined as the performance difference between the worst- perrst-peforming and the best-performing PINN. If a network architecture generalizes well across the range of tasks, both the accuracy and performance disparity should be low. This analysis can also serve as a tool to assess which PDE tasks an architecture struggles with, providing valuable insights for further improvements.
4.3.2 Computational Efficiency
To evaluate computational efficiency, four key metrics are commonly reported. First, the parameter count provides measure of the model size, which impacts memory usage. Second, the number of MACs (Multiply- Accumulate Operations) directly reflects the computational complexity, influencing processing speed. Third, the epoch count assesses convergence by reporting either the final accuracy within a set epoch budget or the number of epochs required to reach a larget error threshold. Fourth, training time offers a direct quantification of computational cost by measuring the duration needed to achieve the desired accuracy. These metrics collectively provide a comprehensive view of the computational demands associated with different PINN architectures and training strategies, facilitating informed decisions for their deployment in resource-cosstrinne environments.
19

---

Published in Transactions on Machine Learning Research (03/2025)
5 Applications & Discussions
Transfer learning and meta-lcarning techniques have shown considerable promise in enhancing the adaptivity of PINNs. These lechniques offer promising solutions for both forward and inverse problems, extending beyond traditional benchmarks to real-life applications in various engineering and scientific domains. This trend highlights how adaptive PINNs are increasingly bridging theoretical advancements with practical utility, as evidenced by the works marked (*) in Tables and
Adaptivity in Forward Problems
In forward problems, efficient adaptivity focuses primarily on adaptation speed and retrieval of previously unseen solutions from similar tasks. This ability is invaluable in applications requiring fast queries within specific task range, such real-time adaptive systems and design optimization. Reducing the number of collocation points and training time is crucial factor in this context. Another promising application of PINN adaptivity is function discovery, where pre-trained PINN, initially trained on an easy-to-solve PDE, is tuned using sparse measurements obtained from experiments or sensors, as demonstrated by Chen
et al.
2021
Adaptivity in Inverse Problems
The idea of adaptive PINNs extends to inverse problems, where they excel at inferring unknown parameters and minimizing data dependencies. In these cases, efficient model adaptivity is key in reducing the number of data samples needed to infer initial or boundary conditions, making PINNs especially valuable for real-time adaptive systems. For example, Xu et al. (2022) demonstrated the effective use of transfer learning in PINNs for real-world problems like tunneling, where the model is pre-trained on an casily solvable task and then fine-tuned using limited real-world data, highlighting their effectiveness in inverse applications.
Comparison with Conventional Solvers
Despite their adaptive capabilities, PINNs must be rigorously evaluated as forward solvers by direct com- parisons with conventional numerical methods. Studies such as Qin et al. 2022 have shown that while PINNs offer a flexible, mesh-free, and data-free approach that is beneficial for complex geometries, they often struggle to compete with highly optimized PDE solvers in terms of computational efficiency. For in- stance, despite leveraging meta-lcarning to accolerate PINN optimization, Qin et al. (2022) found that their meta-solver remained slower than a strong JAX baseline using the finite-volune method with Godunov flux. This highlights the significant speed advantage that conventional PDE solvers, particularly those imple- mented in nigh-performance computing frameworks, can achieve. However, it is important to note that the comparison of Qin et al. (2022) focused a single benchmark problem, and broader testing across diverse scales and PDE types is needed to fully characterize the strengths and limitations of adaptive PINNs versus optimized traditional solvers. Furthermore, other works surveyed here lack systematic comparisons, limiting conclusive insights into the practical viability of PINNs.
Emerging Alternatives: Differentiable Solvers and Active Learning
Beyond adaptive PINNs, emerging fast differentiable solvers, combined with active learning, present an alternative for creating surrogate models tailored to specific solution distributions. A fast differentiable solver could iteratively refine surrogate model by aligning it with desired solution range, using active learning to prioritize critical regions of the domain. In this context, transfer learning and meta-learning techniques could enhance these approaches by enabling efficient adaptation to new tasks or domains, reducing the need for extensive relraining. However, these methods are still being developed, and their broad adoption remains uncertain. Advancing these methodologies will require systematic comparisons to determine their tradc-offs accuracy, generalization, and computational cost, particularly in relation to adaptive PINNs. library (Ataci & Salehipour 2024 ; the work of (Pervez et al. 2024 and (Chen et al. 2024), which combines mechanistic
Some examples of differentiable solvers include XLB, a differentiable
20

---

Published in Transactions on Machine Learning Research (03/2025)
neural networks with differentiable ODE solver to enable interpretable solutions; and Diffrax, JAX-based library for numerical differential equation solvers (Kidger 2021), to name a few. In the context. of neural surrogate models, there is recent work exploring active learning Musekamp et al. (0025). which could also be extended to PINNs.
Benchmarking Challenges and Standardization Needs
Although significant progress has been made, challenges persist that may hinder the continued advancement of adaptive PINNs. One such challenge is the variety of benchmarking strategies used in different studies, which complicates the process of directly comparing methods. Without standardized evaluation frameworks, it becomes difficult to draw meaningful conclusions about the effectiveness of different approaches. In future research, it is necessary to ground these strategies to facilitate easier comparison between different methods. Establishing standardized evaluation guidelines, as proposed in Section 4 is crucial to ensure fair comparisons and evaluating generalization capabilities. The diversity in benchmarking strategies makes it currently difficult to determine the most effective technique. This challenge is reflected in the contradictory conclusions drawn by different works. For example, Penwarden et al. (2023) found that MAML weight initialization only marginally improves performance compared to random initialization, while Qin et al. (2022) and Liu al. (2022) reported opposite findings. Establishing a common ground for evaluating these techniques is essential. Additionally, refining the definitions of terms such as 'indiisribbution' 'out-of- distribution' tasks, and 'related' or 'similar' PDEs can facilitate mcaningful comparisons and help identify suitable applications for each method.
Strategies for Enhancing Adaptivity
Nevertheless, a comparison of adaptivity benefits, using standard PINN the baseline, reveals significant improvements in accuracy and training efficiency when adopting weight initialization methods. The full details of this comparison, including the specific equations evaluated, are provided in Appendix B This comparison underscores the effectiveness of weight initialization approaches in enhancing adaptivity, with the works of Liu et al.] (2022). Penwarden et al. (2023), and Cho et al. (2024b) demonstrating particularly promising results in this regard. These methods enable rapid adaptation by leveraging pre-trained models, making them especially well-suited for tasks that require fast adaptation. However, for these methods be truly effective, their pre- trainig phase must capture broad range of shared representations, ensuring generalizability across different tasks.
Similarly, Lechniques that use basis function expansions, such as those in Chen & Koohy (2024) Cho et al. (2024b), and Gao et al. (2022), offer efficient fine-tuning by reducing the required number of parameters. By restricting the hypothesis space to a smaller set of expressive basis functions, these methods can achiove laster convergence compared to unstructured parameter optimization, provided that the selected bases align well with the task structure. However, like weight initialization methods, shared bases often frozen during fine-tuning must be expressive enough to capture a wide variety of tasks.
An alternative strategy to further enhance adaptivity involves incorporating adaptive basis functions library of pre-trained basis weights, suggested by Mustajab et al. (2024). Combining this approach with a mixture-of-experts framework could significantly boost the model's generalization ability. Furthermore, while explicit basis function methods, such as GPT-PINN (Chen & Koohy] 2024) and One-shot PINN Desai et al. (2021, construct global basis functions, exploring smaller local subdomains within the PDE domain may offer further improvements in both efficiency and task-specific adaptability. These stratogics highlight potential avenues for future research to improve adaptivity and efficiency in PINNs.
In addition to these approaches, another promising method is the self-referential learning technique intro- duced by Huang et al. (2022). This approach achieved training runs nearly nine times faster than traditional PINNs when applied to the medium-difficully Burgers equation and about five times faster for the more com- plicated Maxwell equations. Their results suggest that using task-specific inputs and keeping the weights trainable hold particular promise for OOD scttings, where adaptivity new tasks is crucial. Morcover, as previously discussed, the adaptive loss function approach of [Song al. (2024) is particularly notable for being one of the few methods benchmarked the complex Lid-driven cavity flow equation. This method
21

---

Published in Transactions on Machine Learning Research (03/2025)
moves away from traditional weight initialization techniques by dynamically adjusting error weights through attentional mechanism, demonstrating its potential for challenging fluid dynamics problems. Exploring such innovative loss function strategies could be another promising direction for future research, to further improve optimization and adaptability in PINNs.
Future Research Directions
While recent advancements in adaptive PINNs are encouraging, there remain substantial opportunities for further improvement. Future research should focus on developing optimizers tailored for PINNs and more efficient methods for approximating the derivatives of the PDE-loss components, such as those proposed by (Shi et al. (2024), to accelerate training. Another promising direction is learning multiple tasks together through parametric PINNs, as explored by Cho et al. (2024a), which could enable a shared representation across tasks and improve generalization. Additionally, creating pre-trained weight libraries tailored to spe- cific problem classes may enhance adaptability and reduce computational overhead. Addressing inefficiencies during adaptation, through strategies to mitigate negative transfer learning, could be crucial to robustness. Incorporating advanced techniques, such as gradient-based attention mechanisms or weightttweighhed loss fanctions, might dynamically prioritize critical regions within the domain, enabling PINNs to allocate re- sources more effectively. These advancements are poised to further enhance the adaptability and elficiency of PINNs, making them applicable to broader range of scientific and engineering problems.
6 Conclusion
This study highlights the potential of enhancing the adaptivity PINNs through meta-learning and transfer learning techniques. By enabling the reuse of learned information, these approaches can improve the efficiency of PINNs, particularly in applications where repeated evaluations of similar tasks are required. Instead of solving each PDE from scratch, adaptive PINNs aim to leverage prior knowledge, making them promising direction for reducing computational effort in solving familics of related problems.
The works presented in this survey support the idea of adaptive PINNs, demonstrating progress while also highlighting areas that require further development. Additionally, this survey provides insights into method- ologies for asscssing efficient model adaptation, including relevant metrics and benchmarks. Standardizing evaluation practices is essential for facilitating meaningful comparisons across studies and advancing the field.
Adaptive PINNs have the potential to expand the applicability of PINNs in scientific and engineering prob- lems, particularly in scenarios where limited data is available and rapid evaluations are necessary. However, challenges such as computational overhead and generalization across diverse problem domains remain open challeuges for further investigation. Continued research in optimizing adaptation techniques, loss function design, and multi-task learning has the potential to further advance adaptive PINNs within the broader landscape of computational methods for PDEs.
Acknowledgements
We thank the anonymous TLMR reviewers whose feedback helped to improve the manuscript. We acknowl- edge the support of the German Federal Ministry of Education and Research (BMBF) part of InnoPhase (funding code: 02NUK078), the Deutsche Forschungsgemeinsccaat (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2075 - 390740016, and the Stuttgart Center for Simulation Science (SimTech)
22

---

Published in Transactions on Machine Learning Research (03/2025)
References
Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb. Solving inverse problems using data-driven models. Acta Numerica, 28:1-174, 2019.
Mohammadmehdi Ataci and Hesam Salchipour. XLB: differentiable massively parallel lattice Boltzmann library in Python. Computer Physics Communicatiins, 300:109187, 2024.
Hubert Baty. A hands-on introduction to physics-informed neural networks for solving partial differ- ential equations with benchmark tests taken from astrophysics and plasma physics. ar preprinl arXiv:2403.00555 00599, 2024.
Jonathan Baxter, Rich Caruana, Tom Mitchell, Lorien Y Pratt, Daniel Silver, and Sebastian Thrun. Learn- ing to learn: Knowledge consolidation and transfer in inductive systems. In NIPS Workshop, htlp://plato. acadiou. ccoursescompp coop//silmeerrrr///e/eoo
LTL/transfer workshop, 1995.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synoptic learning rule. Citeseer, 1990.
Rafael Bischof and Michael A Kraus. Mixtuure-fFeepertt-eeeeeell meta-learning for physics-informed neural networks. In Proceedings of 33. Forum Bauinformatik; 2022.
Stevo Bozinovski. Reminder of the first paper on transfer learning in neural networks, 1976. Informatica (Slovenia), 44(3), 2020. doi: (0.31449/TNF. V44T3 2828.. URL http::///o....rr///004444/9 v4413.
2828
José Caballero and Ignacio E Grossmann. An algorithm for the use of surrogate models in modular flowsheet optimization. AICLE journal, 54(10):2633 2650, 2008
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, 37(12))1727-1738, 2021.
Souvik Chakraborty. Transfer learning based multi-fidelity physics informed deep neural network. Journal of Computational Physics, 426:109942, 2021.
Jiale Chen, Dingling Yao, Adeel Pervez, Dan Alistarh, and Francesco Locatello. Scalable mechanistic neural networks. arXiv [cs.LG], October 2024.
Xiaoli Chen, Jinqiao Duan, and George Em Karniadakis, Learning and meta-learning of stochastic advection diffusion diffusion-reaction reaction systems from sparse measurements. European Journal of Applied Mathematics, 32(3): 397-420, 2021.
Yanlai Chen and Shawn Kooby. Gpt-pinn: Generative pre-trained physics-ifformed neural networks toward non-intrusive meta-learning of parametric pdes. Finite Flements Analysis and Design, 228:104047, 2024.
Shijun Cheng and Tariq Alkhalifah. Meta-pinn: Meta learning for improved neural network wavefield solutions. arXiv preprint arXiv:241.11100, 2024.
Woojin Cho, Minju Jo, Haksoo Lim, Kookjin Lee, Dongeun Loe, Sanghyun Hong, and Noscong Park. Pa- rameterized physics-informed neural networks for parameterized pdes. arXiv preprinl arXiv:2408.. 09446, 2024a.
Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork- based meta-learning for low-rank physics-infol informed neural networks. Advances in Neural Information Processing Systems, 36, 2024b.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine lcarning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
Shaan Desai, Marios Mattheakis, Hayden Joy, Pavlos Protopapas, and Stephen Roberts. One-shot transfer learning of infoice neural networks. arXiu preprint arXim:21 2110:100033 1286, 2021.
23

---

Published in Transactions on Machine Learning Research (03/2025)
John Eason and Selen Cremaschi. Adaptive sequential sampling for surrogate model generation with artificial 2014.
neural networks. Compulers 8 Chemical Engineering,
Salah A Faroughi, Nikhil Pawar, Celio Fernandes, Maziar Raissi, Subasish Das, Nima K. Kalantari, and Seyed Kourosh Mahjour. Physics-guided, physics-informed, and physics-encoded neural networks in sci- entific computing, 2023. URL ftttp://aaxxiv.org/ab//////
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Scbastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. arXiv preprint ar 1rXi::8822.0105, 2018.
Yihang Gao, Ka Chun Cheung, and Michael K Ng. Svd-pinns: Transfer learning of physics-informed neural networks via singular value decomposition In 2022 TEEE Symposium Series on Compulational Intelligence (SSCT), pp. 1443-1450, IEEE, 2022.
Somdatta Goswami, Cosmin Anitescu, Souvik Chakraborty, and Timon Rabezuk. Transfer learning enhanced physics informed neural network for phase-field modcling of fracture. Theoretical and Applied Fracture Mechanics, 106:102447, 2020.
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505-$5)) 85100, 2018
Sepp Hochreiter, A Steven Younger and Peter R Conwell. Learning to learn using gradient descent. In 2001: International Conference Vienna, Austria, August 21 21-25, 2001
Artificial Neural Proceedings 11, pp. 87 94. Springer, 2001.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169, 2021.
Xiang Huang, Zhanhong Ye, Hongsheng Liu, Shi Ji, Zidong Wang, Kang Yang, Yang Li, Min Wang, Haotian Chu, Fan Yu, et al. Meta-auto-decoder for solving parametric partial differential equations. Advances in Newral Information Processing Systems, 35:23426-2333 234388, 2022.
Tomoharu Twata, Yusuke Tanaka, and Naonori Ueda. Meta-learning of physics-informed neural networks for efficiently solving newly given pdes. arXiv preprint orXiv:2310.1327) 2023.
Zichao Jiang, Junyang Jiang, Qinghe Yao, and Gengchao Yang. A neural network-based pde solving algo- rithm with high precision. Scientific Reports, 1314),447, 2023.
Yang Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric partial differential equations using the neural convolution. SIAM Journal on Scientific Computing, 43633:A1697-11119, 2021.
Patrick Kidger. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacbarya, Andrew Stuart, and Anima Anandkumar. Fourier ncural operator for parametric partial diffcrential equations. arXiv preprint Xrxi:22010. 088995, 2020.
Shuning Lin and Yong Chen. Gradient-enhanced plysics-informed neural networks based on transfer learning for inverse problems of the variable coefficient differential equations. Physica D: Nonlineor Phenomena, 459:134023, 2024.
Xu Liu, Xiaoya Zhang, Wei Peng, Weien Zhou, and Wen Yao. A novel meta-learning initialization method for physics-informed neural networks. Neural Computing and Applications, 34(17):14511 14534, 2022.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Decponet: Learning nonlincar operators for identi- fying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03190 2019.
24

---

Published in Transactions on Machine Learning Research (03/2025)
Daniel Musekamp, Marimuthu Kalimuthu, David Holzmüller, Makoto Takamoto, and Mathias Niepert. Active learning for neural PDE solvers. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. URL
http://0ab/2000005666 01536
Abdul Hannan Mustajab, Hao Lyu, Zarghaam Rizvi, and Frank Wuttke. Physics-informee neural networks for high-frequency and multi-scale problems using transfer learning. Applied Sciences, 14(8):3204, 2024.
Geoffrey Négiar, Michael W Mahoney, and Aditi S Krishnapriyan. Learning differentiable solvers for systems with hard constraints. arXiv preprint arXiv:2207.0800 088677, 2022.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009.
Raphaäl Pellegrin, Blake Bullwinkel, Marios Mattheakis, and Pavlos Protopapas. Transfer learning
with dlysics-informee neural networks for efficient simulation of branched flows ar preprint arXiv:2211.00214 2022.
Wei Peng, Weien Zhou, Jun Zhang, and Wen Yao. Accelcrating physics-informed neural network training with prior dictionaries. 2020.
Michael Penwarden, Shandian Zhe, Akil Narayan, and Robert M Kirby. A metalcarning approach for physics- informed neural networks (pinns): Application to parameterized pdes. Journal of Compulational Physics, 477:111912, 2023.
Adeel Pervez, Francesco Locatello, and Efstratios Gavves. Mechanistic neural networks for scientific machine learning. arXiv fcs. February 2024.
Konstantinos Prantikos, Stylianos Chatzidakis, Lefteri H Tsoukalas, and Alexander Heifetz. Physics-informe informed neural network with transfer learning (tl-pinn) based on domain similarity mcasure for prediction of nuclear reactor transients. Scientific Reports, (111:466,,, 2023.
Lorien Y Pratt, Jack Mostow, and Candace A Kamm. Direct transfer of learned information among neural networks. In Proceedings of the ninth National conference on Artificial intelligence- Volume 2, pp. 584-589, 1991.
Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. Meta-learning pinn loss functions. Journal of computational physics, 458:111121, 2022.
Tian Qin, Alex Beatson, Deniz Oklay, Nick McGreivy, and Ryan Adams. Meta-pde: Learning solve pdes quickly without mesh. arXiv preprint arXiv:2911.0160, 2022.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the mela-mela-... hook. PhD thesis, Technische Universität München, 1987.
Nocl E Sharkey and Amanda JC Sharkey. Adaptive generalisation. Artificial Intelligence Review, 7(5): 313 328, 1993.
Zekun Shi, Zheyuan Hu, Min Lin, and Kenji Kawaguchi. Stochastic taylor derivative estimator: Efficient amortization for arbitrary differential operators. arXiv preprint arXiv:2412.0000 1000088 2024.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differ- ential equations. Journal of Computational Physics, 375:1339- 2018.
Yanjie Song. He Wang, He Yang, Maria Luisa Taccari, and Xiaohui Chen. Loss-attentional physics- informed neural networks. Journal of Computational Physics, 501:112781, 2024.
25

---

Published in Transactions on Machine Learning Research (03/2025)
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pfttiger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Newral Information Processing Systems, 35:1596 1611, 2022.
Kejun Tang, Xiaoliang Wan, and Chao Yang. Das-pinns: A deep adaptive sampling method for solving high-dimensional partial differential equations. Journal of Computational Phaisic,, 476:111868, 2023.
Maryam Toloubidokhtii, Yubo Ye, Ryan Missel, Xiajun Jiang, Nilesh Kumar, Ruby Shrestha, and Linwei Wang. Dats: Difficult--wwat task sampler for meta-learning physics-informn informed neural networks. In The Twelfth International Conference Learning Representalions, 2023.
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non-adaptive and residual- based adaptive sampling for physics-informet informed neural networks. Computer Methods in Applied Mechanícs and Engineering, 403:115677 2023.
Chen Xu, Ba Trung Cao, Yong Yuan, and Günther Meschke, Transfer learning based ilyysies-iffrmed neural networks for solving inverse problems in tunneling, arXiv e-prints, pp. arXiv 2205, 2022.
Jeremy Yu, Lu Lu, Xuhui Mong, and George Em Karniadakis. Gradient-onhaneed physics-informed neural networks for forward and inverse pde problems. Compuler Melhods in Applied Mechanics and Engineering, 393:114823, 2022.
Dongkum Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physics- informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, November 2019. ISSN 0021-9991. doi: 10.1016/j.iee (t000//j.pp.211.077. 2019.07.048. URL http: //dx.
doi. www.oogggg.. 10t///0.1116... 10l.1001/j.jjr jep. 2019. 07 .048
Linlin Zhong, Bingyu Wu, and Yifan Wang. Accelerating physics-informed neural network based 1d are simulation by meta learning. Journal of Physics D: Applied Physics, 56(7):074006, 2023.
Meijun Zhou and Gang Mei. Transfer learning baasidg-based coupling of smoothed finite element method and infsree-informed neural network for solving elastoplastic inverse problems. Mathematics, (1(1))22529, 2023.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing Hc. comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1))43376, 2020.
26

---

Published in Transactions on Machine Learning Research (03/2025)
A Benchmark Equations
A.1 Equation Classification
Table 4 synthesizes the broader landscape of benchmark usage, categorizing partial differential equations (PDEs) by three critical dimensions: computational complexity, key technical challenges, and adoption frequency the surveyed literature. This structare enables systematic comparison of problem difficulty and helps researchers select appropriate benchmarks for specific evaluation needs.
The "Complexity" column rates equations from 1 to 5 stars (Ix to putational demands. For example, the Poisson equation (1 star) serves a low-complexity baseline due to its linear, stcady-state nature, while the Navier-Stokes equations (5 stars) represent the upper complex- ity extreme, requiring resolution of coupled velocity- fields with nonlinear advection and complex boundary conditions.
reflecting relative com-
Key Drivers of Complexity identify the primary technical challenges for each PDE:
Time dependence: Critical in transient problems (e.g., Burgers', Wave equations)
Nonlinearily: Dominates shock-forming systems (e.g., Allen-Cahn, A-D-R)
High-frequency oscillations: Challenges resolution limits (e.g, Schrödinger, Helmholtz)
Multi-cariable coupling: Increases system dimensionality (c.g., Navier-Stokes)
The "Used By" column quantifies benchmark prevalence in this survey, revealing distinct adoption pat- terns. Burgers' equation emerges as the most popular intermediate benchmark (9 studies), offering balanced complexity through its nonlinear shock dynamics. In contrast, high-complexity systems like A-D-R and Navier-Stokes appear only once each, reflecting their specialized computational requirements.
A.2 Poisson Equation
The Poisson equation is a second-order elliptic PDE appearing in many fields, such as electrostatics, steady heat transfer, and many others. This equation has the following form:
- -Au(w) ()) = f(x),
x E Ω
u(x) (x) = 0, x E an,
where A(-) is the Laplace operator. A common feature among all works is that the domain is 2D, specifically Q C [] x [-,, except for Bischof & Kraus (2022). which uses L-shaped domain. The forcing source term f(x) changes among works:
A.3 Burgers' Equation
Burgers' equation is Line-dependent PDE that models a system consisting of moving viscous fluid. The ID form of the equation models the fluid flow through an ideal thin pipe. The Burgers' equation is given by:
Ou
Du
x EEΩ,
LE [0,
0t
Or
dr
(10)
u(x, 1) = O,
x E nn I [0,T],
u(x, un(x),
x E Ω
The unknown u(x,1) is the speed of the fluid, and the fluid viscosity. When the viscosity is low, then the fluid flow develops shock wave.
Most of the works presented here treat as the parameter that defines a task, the initial condition uo(x) (ix() and the computational domain as Ω []1 e [0, 1]. The table below shows the different choices of u across various works.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Complexity of Different Models
Model Complexity
Model
Used by Complexity
Key Drivers of Characteristics Complexity
Poisson
5
Goometric/boundayy complexities
Lincar, steady-state, single scalar field with fixed boundary values
Helmholtz
2
High-
Linear, oscillatory, high wave numbers require fino-scale reso- lution
(requencyosccilatto
solutions
Wave
1
Time dependence
propagation and behavior over time
Describes
Burgers'
9
Nonlinearity, Time dependence, Shocks
Nonlinear advection, shock for- mation (low viscosity), sharp gradients
Allen- Cann-
3
Nonlinearity, Time
Nonlinear reaction-diffusion for phase transitions; stiff, time- dependent dynamics
dependence
Schrödinger
3
High- frequency /oscillatroy solutions
Complex-valued solutions, time- dependent, possibly nonlinear
A-D-R
Nonlinearity, dependence, Stiffness
Time
Combines advection, diffusion, and nonlinear reaction; stiff gra- dients Coupled PDEs (velocity and pressure), nonlinear advection, complex boundaries (sharp cor- ners)
Navier-Stokes (LDC)
1
Nonlinearity, Time dependence, Multi- variable
Note: Abbreviations: Adveecton-Diffuuson--lecccio = A-D-R, Lid-driven Cavity LDC.
Table 5: Different forms of f(x,y) y) used in various studics. Desai et al. (2021 employs different forcing term during testing. the work Liu et al. (2022). n represents the number of heat sources, and U denotes uniform sampling.
Literature
f(x, v)
Parameters
Desai et al. (2021
sin skm()) sin (kry)
E (1,2,3,4)
Em 0a-ex0 exp (- ((-a)))+(y+bb))"
a4, a; ~ 1(001,0.9)
0.01
Liu et al.
2022
G ~ 4(0.8, 1.2)
Bischof & Krans
2022
1
Song et al.
2024
2 2 sin (Tx) sin(Ty)
28

---

Published in Transactions on Machine Learning Research (03/2025)
Table 6: Different choice of vfor Burgers' Equation used in various studies.
Literature
Parameters
Liu et al. (0022)
[0, 0.1/m]
Penwarden et. al.
(2023
M E [0.005, 0.05)
Chen & Koohy
(2024)
[0.005, 1/m]
Toloubidokhti ct al.
(2023)
E [0.001,0.1]
Chen & Koohy
(0244)
[0.005, 1/mr
Psaros et al.
2022
E [0.001,0.002] & v E [0.01, 1.0)
Song et al.
2024)
0.01/T
A.3.1 Allen-Caie Cahn Equation
The Allen-Cahn equation is given by:
du
XAu + e(u e(u"= = f(x,
-u)
E [0, I E Ω x E in, t [0,
01
(11)
u(x,t) () 1,
(( 0) x²² cos(
where Ω = [- represents the spatial domain, and
denotes the final time. The coefficient. À is chosen from the interval [0.0001, 0.001], while the parameter E, which controls the strength of the nonlinear term, is selected from the range 5]. The forcing term f(x,t) is set to zero for this study, focusing on the intrinsic dynamics of the Allen-Cahn equation. This represents per Chen & Koohy (0244). An alternative formulation of the IBVP exists in other works that derive forcing term based on an exact solution, such as Penwarden el al. 2023 and Xu el. al. 2022
initial-boundary value problem (IBVP)
A.4 Wave Equation
The wave equation for scalar wave function u(x, 1) is given by:
pu V
(12)
0t
where is the wave speed and 2 is the Laplacian operator in three dimensions.
A.5 Helmholtz Equation
The Helmholtz equation for scalar field u(r) is given by:
Au + k2 = O.
(13)
where k is the wave number related the wavelength
A.5.1 Schrödinger Equation
The time-dependent Schrödinger equation for a single particle in three-dimensionall space is given by:
ih Ov(r,) 12 ( V(r)
(14)
Øt 2m
where (r, t) is the wave function, r (a, 2) are the spatial coordinates, is time, h the reduced Planck's constant, m is the mass of the particle, 2 the Laplacian operator, and V(r) is the potential energy function.
29

---

Published in Transactions on Machine Learning Research (03/2025)
A.6
diveecioorreactionnddfffouo equalions, considered in this section, are known to be stiff problems when the advection term dominates over the diffusion one. In such cases, sharp transition layers appear in the solution, which are difficult to capture by traditional numerical schemes. Baty 20224) The advection-reattion-dlffisooo equation is given by:
du Ōt
+ Vu DV R(u),
(15)
where == u(, t) is the dependent variable (scalar field), is time, (æ, y, z) represents spatial coordinates, v (Va, Uyy U2) is the velocity field (advection term), D is the diffusson coefficient, 4 is the Laplacian operator, and R(u) is the reaction term.
A.6.1 Lid-driven Cavity Flow
The lid-driven cavity flow equations are:
du de
0,
dr Oy
u Du
du
Op
v
+ vv + Fe.
(16)
dx
dy
p dx
dv
dv
10p
+ + Fy.
dr
Dy
pdy
with boundary conditions:
u(2, 0) = 0, u(x, 1) = (lid),
v0, v) = v(1, v) = (walls).
u(x,) y) v(x 0) v(x, 1) (other boundaries).
Here, u(x, v) and v(z, u) are the velocity components, p(u, y) the pressure, P the fluid density, v the kinematic viscosity, and Fz, Fy are additional body forces.
B Method Comparison
This table provides a comparison of various PINN methods across different problem types, training ap- proaches, and improvements in accuracy and speed. It highlights the accuracy improvement (%) and speed- up/slowdown (%) achieved by each method compared to the baseline PINN. The table includes key details such the author, short name, problem type, and training strategy (pro-train cetrrannffinn-tie /fine-tune), along with the data type used in each case. It also reports the end accuracy (for both the PINN and method) and the number of epochs and time required for training. The improvement percentages are calculated based on the following formulas:
PINN PINN
accuracy Methodaccuracy x 100
Accuracy Improvement (%)
PINN PINN MaNNacuurcyy
PINNepochs
PINN epochs
-1) 100
Speed Up/Slowdown (%)=
Methodepactus
Where speed-up calculated in terms of either the number of epochs or total training time (* denotes training time).
30

---

Published in Transactions on Machine Learning Research (03/2025)
In terms of Table the following can be observed: TL-gPINN (Lin & Chen 2024
enhances accuracy over 2022) outporforms PINN in the
gPINN but remains slower than the baseline PINN. Reptile QLiu et al. Burgers' inverse problem, achieving faster convergence with fewer epochs. SVD-PINN (Gao et al. 2022) demonstrates a negative performance compared to the standard PINN for the Allen-Cahn equation. Curricu- lum Mustajab et al., 2024] accelerates convergence by 50%, though final accuracy is comparable PINN. Interpolation (Penwarden et al., 2023) improves convergence speed through superior weight initialization but does not significantly boost accuracy. GPT-PINN (Chen Kooby, 2024] converges more quickly but does not surpass PINN in final accuracy, potentially due to the limited expressiveness of its basis functions. MAD-PINN (Huang et al. 2022) offers 30% accuracy improvement and 362% speed-up over PINN for the Maxwell equation, & complex problem. LA-PINN (Song et al. 2024) meta-learns the loss function, out- performing PINN on both simple and complex equalions, though the performance gains are less pronounced for more difficult problems. Hyper-Ir-PIN (Cho et al.) 2024b) excels in both accuracy and speed for the Helmholtz equation, although the speod incrcase may stem from the similarity between the target and source tasks.
Note: The values presented in the table approximated, some were extracted from graphical data.
31

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4 Plg. 11
page Fig.4 (a)
pagge 8 Fig.4 (b)
Table 6 Fig. 17
Tabble 5 19g.15
Page Fig.8
Table 21 Fig.7
Reference
Fig-10-top
Table 7: Comparison of PINN methods. The reference columns refer to the location of where the values were extracted from the source works.
Table:
Table
Fig.
Fig.0-
Flg.
Table
Table
Table
Fig.13
Flg.19
Time
Meshod
1862"
2203*
500
20 m00
1000
300
3000
26
30
44
350
000
8000
10
20
13
Epochs
PINN
302*
000
8000
1500
4000
2000
30000
146
178
469
12000
10-2 3000
60000
10000
10,000
100000
2000
60
Method
3.40 x 10 1.91 10
3
8,50 x 10
10
10-7
10-5
1.34 10
4.50 x 10
1.60 x 10
1.13 x 10
10-3
x 10
10-3
Eadd Accuracy
5.00
6.47
8.50
1.00
0.028
2.83
2.29
0.0285
11.8
PINN
9.00 x 10
5.00 x 10
10-3
8 10-10-3
6.90 x 10-10--
5.10 x 10-10--
5.50 0 10
1.45 10
10-1
1.00 x 10" 22-2
6.87 x 10
2.87 10-4
3 33 x 10
Abbreviations N/S = Not Specified. Negative percentages indicate worse performance compared to baseline PINN.
3.00 x
56.91 01.91
0.04
1.00
Schridingo (Nonlinear)
Schrödinggr (Lincar)
A Caln
Schrödinger
Allen Cala
Col. Points Improvement (%6) Equattin
Burgers
Polsson
Burgers
Durgess
Burgers
Burgers
Maxwells
Burgess
Poisson
Helmholta
Helmbolta
Waxe
Heat
Speed
000
660
507
101
462
193
DGG
757
302
000 900
19 900
12
Accuracy
44
71
-844
0
98
12
16
21
30
90
98
Weight Init 2000, 200
Curricalum 40000
Curriculum 40000
Weight Init N/S
Curriculum 512
Weight Init 4000
Weight Init 2000
Weightt Init 2000
Weight Init 10100
10100
10100
20200
N/S
N/S
10300
20400
11400
Weight Init
Weight Init
Weight Init
Network
Problem Type Type
Input
Input
Loss
Loss
Loss
Inverse
Irverse
Inverse
Phrward
Forward
Forward
Forward
Forward
Forward
waadd
Forward
Forward
Forward
Forward
Furward
Forward
Forward
ward
Hyper-LEE LR- PINN
Interpolations
Short Name
MI-gPINN TI-EPINN
SVD-PINN
Curriculum
Interpolations
Interpolations
GPT-PINN
MAD-PINN
MAD-PINN
LA-PINN
LA-PINN
LA-PINN
Reptile
Reptile
Reptile
Reptile
2024
2023
20223
2024
2024
20 22
CNO + W 2022
OF AL
2322
2022
CONETTA den CF
den CE AL
Kooky
2022
TORTI OF W 2022
2024
2024
CI CL 2022ED
Lin chen
PAM AL
SLD
Author
&
SOOIE OF
32


Reviews:
[{"review_id": "LyHjFZrpQw", "reviewer": "Reviewer_m2K5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In the remaining review I mainly focus on sections 3, 4 and 5. Since this is a review paper, for evaluation I judge whether for each reviewed work the reader is able to get  a high-level view of what is done, why and how it was done and what was the conclusion. Opaque descriptions that provide no insight should be avoided.\n\nMeta-learning history box: \n\nSince a number of methods later described depend on MAML there should be a description of how MAML works. Currently the description only says that it “aimed to provide a good weight initialization”. Later there is an indirect reference to MAML’s bi-level optimization and second order derivatives in the context of the Reptile algorithm. This should be restructured.\n\nSection 3.1.1.\n\nThe TL-PINN description is good. It might be useful to also include the source tasks.\n\n The 3rd paragraph mentions TL-gPINNS as an extension of gPINN, but I can’t find any earlier mention of gPINN  nor is there any citation. The description should introduce gPINNs first and then the extension.\n\nThe description of S-FEM method is not clear about the motivation of the method. What is the significance of the FEM data pre-training? And how is it different from other types of pre-training?\n\nSection 3.1.2\n\nThe description of the method from Desai et. al. (2021) is unclear in how it relates to the PINN objective. What is the relation of the ‘final solution’ as it is given in equation 8 to the PINN objective from section 2.2 and equations 2 and 3? Is \\Phi_\\theta the same as u_\\theta? I think the description should use the PINN notation from section 2.2.\n\nThe third paragraph (page 10) retraining for each displacement step is mentioned but is unclear what a displacement step is.\n\nThe description of the methods from Goa et al (2022) and Pellegrin et. al. (2022) are good.\n\nIn the last paragraph discussing Xu et al. (2022) what are the offline and online stages? Are these the same as pre-training and fine-tuning or something else? Is there streaming data involved?\n\nSection 3.2.1\n\nI think this section should first include a general description of how the meta-learning initialization techniques are applied to PINNs. Without such a general description the individual methods are very opaque and it is not clear how the methods work.\n\nThe third paragraph mentions that the implementations of Zhong et al and Cheng & Alkhalifah differ without describing how they are different.\n\nThe fourth paragraph mentions LEAP without a description of LEAP.\n\nThe fifth paragraph mentions Penwarden et al. comparing several models. What were the conclusions of this comparison and the two-step initialization?\n\nIn the last paragraph what were the conclusions from Cho et al?\n\n3.2.2, 3.2.3, 3.2.4 have good descriptions of the relevant methods. However, one paragraph at the beginning summarizing the approaches would be helpful such as the one at the beginning of section 3.2.4.\n\nSection 3.3\n\nI think the sentence “Few-shot learning aims to minimize the number of training examples required…” should be reconsidered, since few-shot learning is more than just sample efficient learning.\n\nThe last two paragraphs are not very informative about the works discussed. For instance, what is the method of Mustajab (2024) and how does it achieve few-shot learning? Similarly for the other methods there is not sufficient discussion.\n\nSections 4.1, 4.2\n\nThere is no discussion of the character (difficulty,complexity, scale etc) and comparison of the various benchmark problems. Similarly there is no discussion on the usefulness or otherwise of the various error metrics that are used in the literature.\n\nSection 5.\n\nThe discussion on application is also very sparse and only quotes various applications from the papers. I think it is missing a more critical evaluation of the types of applications for which adaptive models are useful and those which they are not.\nA full illustrative example of the benefits of adaptivity would be very useful and would make the discussion concrete.\n\nTypos/Grammar:\n\nTransfer learning box: “Sharkley”\n\nTransfer learning box: “Pan” reference missing.\n\nTransfer learning box: “Zhuang et al.” citation format.\n\nMeta learning box: “work of Schmihuber (1987), work”\n\n2.6: ‘Burgers’ apostrophe\n\n3.2: ‘study of this techniques’\n\n3.2.1: ‘adoptation’\n\n3.2.1: page 12 ‘...Xavier in a unsupervised…’", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a review of adaptive physics informed neural network models. Physics informed networks are now well-known models for solving forward and inverse problems involving partial differential equations. However, they are limited by the need for reoptimization with changes in PDE parameters.\nThis indicates a need for models that can adapt to changes in parameters and this paper reviews a number of such models.\n\nThe paper is well organized. It begins with an introduction of PDE methods dividing them into traditional solvers, neural PDE solvers and neural surrogates (such as neural operators), with the advantage of neural PDE solving (such as PINNs) being that they include physical laws in learning. However PINNs are not naturally adaptive.\n\nSection 2 of the paper provides an overview of physics-informed networks and related them to classical collocation methods. Section 2.4 also discusses a classical adaptation method, reduced order modeling and discusses their relation with PINNs. Section 2.7 and 2.8 provide a brief introduction to transfer learning and meta-learning.\n\nSection 3 comprises the core of the paper where various transfer learning and meta learning methods for PINNs are discussed. The transfer learning methods are divided into methods with full fine-tuning and methods with parameter-efficient fine tuning with a brief section on curriculum learning. Meta-learning methods follow the taxonomy of Hospedales et al (2021), dividing the models into 1) models that learn the weight initialization, 2) models that learn the network structure, 3) models that learn the loss function, 4) models that learn the input and 4) few-shot learning models.\n\nSection 4 presents benchmarks and metrics; section 5 has a discussion of applications and the paper ends with a conclusion in section 6.\n\nIn my opinion the paper presents a useful overview of the state-of-art for adaptive physics-informed networks. There are however some concerns of clarity in discussion of some referenced works (see below), a full discussion of benchmarks and applications is missing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The concerns mentioned above regarding clarity and missing discussion in referenced works should be addressed. \n\nFurthermore the sections on benchmarks, metrics and application need significant changes. I think that a critical discussion of applications is important for this paper.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3QzgeS0nW", "reviewer": "Reviewer_otXa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Overall, the topic and the work are well-structured and easy to follow. \n- Most concepts are introduced to a sufficient level of detail.\n- The survey seems comprehensive/exhaustive (although I would like the authors to clarify this in the rebuttal and the manuscript).\n- Sections 2.3 - 2.5 on the adaptability in classical reduced-order modeling are an original perspective in the PINN literature.\n- Altogether, this serves as a valuable entry point to adaptive PINNs.\n\n**Weaknesses**\n- My primary criticism of this work is that it merely collects, groups, and summarizes existing work. Although this is one function of a survey paper, I think the authors overlook an opportunity to offer more to the reader with a stronger discussion. For example, while model adaptivity is motivated repeatedly throughout the work, there is no critical discussion or limitations of the individual works or adaptive PINNs overall. There are no comparisons (qualitative or quantitative) or recommendations on which methods to choose and when. The future works are mostly repeated from the individual papers missing the opportunity to summarize where the entire field is or should be going. In my view, the discussion in Section 6 is currently much too short and superficial, especially given that the paper is 17 pages long.\n- Connecting this criticism to the contributions listed by the authors on page 3, I support 1 (overview) and 2 (survey) as notable contributions, but 3 (metrics and benchmarks) and 4 (future directions) are not significant enough to be distinguished from 2.\n- There are quite a few presentation issues in terms of style and grammar, but also mathematical errors and missing or confusing explanations. However, I expect most of these to be fixed (see the list below).\n- This might be due to phrasing, but some claims are borderline unjustified (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript surveys adaptability in physics-informed neural networks, focusing on transfer learning and meta-learning. After providing an introductory overview of the relevant concepts, the survey collects, summarizes, and groups the relevant PINN literature in several transfer and meta-learning categories. Additionally, sections 4 and 5 briefly discuss metrics, benchmark problems, and applications used and addressed in the adaptive PINN literature.\n\nThe main contribution of this work is the survey itself -- there is no new evaluation or significant interpretation of the existing body of work. The contributions are assessed in more detail below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Errors and suggestions**:\n- Throughout the manuscript you repeatedly refer to \"data generation\". I couldn't understand what exactly this means in the different contexts. It seems it is sometimes used as a synonym for a \"solver\". I would strongly recommend against such a use or, if I misunderstand, to explain more precisely what you mean by \"data generation\".\n- \"data resources\" -> data requirements (page 2)\n- (optional) \"To the best of our knowledge, there has not been a review paper addressing model adaptivity in PINNs.\" This is a reviewer-pleasing statement which I would exclude from the final version.\n- Eq. 3: the residuals are missing the squares (or some norm in general). Furthermore, why are the residuals in bold? These are just scalars (there is some inconsistency with eq 2). Lastly, the statement that \"weights act as a regularization term\" is wrong.\n- Also, the statement \"optimizing ... leads the solution $u_\\theta$ to converge toward the true solution\" is not true -- there is no such convergence guarantee. I get what you mean, but I'd recommend being more precise here.\n- (optional) Eq 6: consider defining what $\\omega(x)$ is (presumably a positive scalar function over the same domain).\n- \"either continuous or piecewise\" (page 6). Piecewise basis functions can and almost always are continuous. Perhaps you mean globally vs locally supported functions.\n- (optional) You introduce POD as Proper Orthogonal Decomposition only after you have used the abbreviation POD several times.\n- Eq 7: you must start the summation at $i=1$ to retain $N$ modes (otherwise it's $N+1$).\n-  (optional) Sec. 2.6: Please define computational efficiency similar to how you did with data efficiency. Does this refer to pre-training, fine-tuning, or also inference?\n- (important) PEFT: in the broader ML community PEFT was developed and refers to fine-tuning of truly large models. I would not recommend referring to any kind of partial weight fine-tuning as PEFT. I would sincerely recommend renaming this section/method or at the very least clarifying what you understand under PEFT.\n- Page 10: \"leveraging on knowledge\" -> \"leveraging the knowledge\"\n- Sec 3.2 first paragraph has multiple errors.\n- Page 11, last sentence \"A\" -> \"The\"\n- Page 12: \"reptile\" -> \"Reptile\".\n- Page 12: \"a SVD\" -> \"an SVD\"\n- Page 12: \"future work will address\" ->  \"future work should address\". Similar issues in other paragraphs (e.g. \"future research is concerned with\")\n- There are many more small errors, so please run the entire manuscript through a writing tool.\n\n\n**Problematic claims**:\n- \"This characteristic poses a barrier to the widespread adoption of PINNs across scientific and engineering applications.\" While adaptability certainly is a barrier, there are other major problems with PINNs (which are also discussed in the introduction). This phrasing can be misinterpreted as adaptability being the primary issue. (similarly with \"this survey seeks to identify promising directions for future research to enable the widespread adoption of PINNs\")\n- \"these methods can help overcome the convergence challenges typically associated with PINNs\" Convergence on complex problems is one of the central issues of PINNs, which has motivated lots of research on this. I believe this is a very bold claim, and while the surveyed papers indicate some potential, I would expect a more critical assessment of this claim, either in isolation or in relation to other approaches that deal with convergence.\n- \"synthesizing insights to establish a foundation for efficient data generation strategies tailored to PINNs\" Similar to the major criticism, I do not think this survey succeeds at synthesizing new insights, certainly not foundational ones. A slight paraphrasing could fix this. \n- \"While both neural operators and PINNs have strengths and limitations, this study suggests that PINNs are more suitable for data generation tasks in scientific and engineering domains, especially in applications with limited data resources.\" Nothing in the main body of text discusses NOs vs PINNs, certainly not to an extent where informed suggestions could be made. I'd suggest to paraphrase this.\n- \"To achieve efficient model adaptation, this work suggests the application of transfer learning and meta-learning to physics-informed neural networks.\" Similar to the above points. It *surveys* the application of ...\n\n\nThe primary quality criterion of TMLR is how well-supported the claims of the submission are. I sincerely invite the authors to assess this feedback and consider reformulating the phrasing where necessary to more accurately reflect their intent and the scope of the work.", "broader_impact_concerns": "This work does not require an ethics statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bJobtla3O2", "reviewer": "Reviewer_g4RT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: perhaps the biggest drawback of vanilla physics-informed neural networks, as pointed out in the paper,  is that they are trained on single PDE instances and have to be retrained from scratch when the PDE changes.  For most scientific and engineering applications this problem makes PINNs impractical. Meta-learning and transfer learning are certainly a promising way to overcome this issue and to the best of my knowledge there is no comprehensive survey that puts all the methods together. Furthermore, I find the PINNs literature particularly difficult to navigate due to the many papers that are published on many different journals/conferences spanning different fields. Therefore, surveys and literature reviews can be very valuable. \n\nWeaknesses: Generally, while such a collection of relevant works can be valuable for the reason discussed above, it can be of little practical utility for the practitioner since it does not directly compare performances of the methods described. At least those that are comparable. With PINNs it is generally hard to get an idea of what works best and for what setting. A truly helpful survey facilitates the job of practitioners when faced with the choice of what algorithms to use.\n\nMinor comments: I don't fully understand the purpose of Section 2.3 and the connection to numerical methods that is described. Could you explain better the radar chart? I don't understand what the values are.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a survey on Physics-Informed Neural Networks that make use of transfer learning and meta-learning algorithms for efficient adaptation to reduce the cost of retraining. The survey begins with a short introduction to PINNs and IBVPs and the similarities between PINNs and numerical methods for solving PDEs. The introduction is followed by a brief description of model adaptivity in Machine Learning focusing on transfer learning, and meta-learning techniques, in particular, parameter efficient fine-tuning and MAML. The core of the paper is an exhaustive list of applications of transfer learning and meta-learning techniques to PINNs aimed at reducing computational cost and improving convergence. The paper ends with a section dedicated to metrics and benchmarks for evaluating adaptivity and efficiency of PINNs, and a section on applications that covers forward and inverse problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I understand the difficulty of comparing performances of PINNs but such a comparison, perhaps highlighting failure modes of the various methods, would certainly add value to the survey that, as it is, represents a valuable and comprehensive list of techniques that may be useful for the newcomer but might be of little use for the practitioner.\n\nCould you add some references to section 2.4?", "broader_impact_concerns": "There are no concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A cataloguing of various techniques is described, including fine-tuning, meta-learning, transfer learning, learning the loss function, the inputs and so on. Metrics and benchmarks are discussed in detail. The text is very clearly written, and the total amout of references is in the order of 70. All reviewers agree that this is a good submission, and while initially they were sceptical, they also upgraded their scores.

Reading the paper, I noticed that some recent published works on 'Mechanistic Neural Networks' are not mentioned. I think they are quite relevant and mentioning them in the surve would make sense.

https://dl.acm.org/doi/10.5555/3692070.3693713
https://openreview.net/forum?id=Oazgf8A24z

All in all, I think this is an important survey that is currently missing from the literature.
Decision: Accept (poster)