DATS: Difficulty-Aware Task Sampler for Meta-Learning Physics-Informed Neural Networks
Authors: Maryam Toloubidokhti
Yubo Ye
Ryan Missel
Xiajun Jiang
Nilesh Kumar
Ruby Shrestha
Linwei Wang
Venue: ICLR 2024 poster (2024)
Keywords: Meta Learning
Physics Informed Neural Network (PINN)
Adaptive Sampler
Partial Differential Equations
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Advancements in deep learning have led to the development of physics-informed neural networks (PINNs) for solving partial differential equations (PDEs) without being supervised by PDE solutions. While vanilla PINNs require training one network per PDE configuration, recent works have showed the potential to meta-learn PINNs across a range of PDE configurations. It is however known that PINN training is associated with different levels of difficulty, depending on the underlying PDE configurations or the number of residual sampling points available. Existing meta-learning approaches, however, treat all PINN tasks equally. We address this gap by introducing a novel difficulty-aware task sampler (DATS) for meta-learning of PINNs. We derive an optimal analytical solution to optimize the probability for sampling individual PINN tasks in order to minimize their validation loss across tasks. We further present two alternative strategies to utilize this sampling probability to either adaptively weigh PINN tasks, or dynamically allocate optimal residual points across tasks. We evaluated DATS against uniform and self-paced task-sampling baselines on two representative meta-PINN models, across four benchmark PDEs as well as three different residual point sampling strategies. The results demonstrated that DATS was able to improve the accuracy of meta-learned PINN solutions when reducing performance disparity across PDE configurations, at only a fraction of residual sampling budgets required by its baselines.

Published as a conference paper at ICLR 2024
DATS: DISHICULTY-AWARE TASK SAMPLER FOR META- LEARNING PHYSICS-INFORMED NEURAL NETWORKS
Maryam Toloubidokhti", Yubo Ye 2 Ryan Missel', Xiajun Jiang', Nilesh Kumar', Ruby Shrestha' & Linwei Wang'
'Rochester Institute of Technology, Rochester, NY, USA ZZhejiang University, Hangzhou, China (mt 6129) @ritr..iee edu, (22230131) @zju ju. edu.cs en
ABSTRACT
Advancements in deep learning have led to the development of physics-infommi neural networks (PINNs) for solving partial differential equations (PDEs) without being supervised by PDE solutions. While vanilla PINNs require training one network per PDE configuration, recent works have showed the potential to meta- learn PINNS across a range of PDE configurations. is however known that PINN training is associated with different levels of difficulty, depending on the underlying PDE configurations or the number of residual sampling points available. Existing meta-learning approaches, however, treat all PINN tasks equally. We address this gap by introducing novel difficulty-aware task sampler (DATS) for meta-learning of PINNs. We derive an optimal analytical solution to optimize the probability for sampling individual PINN tasks order to minimize their validation loss across tasks. We further present two alternative strategies to utilize this sampling probability to either adaptively weigh PINN tasks, or dynamically allocate optimal residual points across tasks. We evaluated DATS against uniform and self-paced task-sampling baselines on two representative meta-PINN models, across five benchmark PDBs well as three different residual point sampling strategies. The results demonstrated that DATS was able to improve the accuracy of meta-learned PINN solutions when reducing performance disparity across PDE configurations, at only fraction of residual sampling budgets required by its baselines
INTRODUCTION
Partial differential equations (PDEs) underpin a broad range of scientific simulations such as fluid dynamics, heat transfer, and electromagnetics (32). While traditional methods for solving PDEs can be numerically and computationally challenging over high-dimensional and complex domains (2)
these challenges are being addressed by recent advancements in deep learning (33) Particularly of note are the physics-informet informed neural networks (PINNs) that integrate the governing PDEs as a loss function to train neural network, approximating PDE solutions without discretizing the underlying solution domain while improving efficiency (30). Because the PINN is supervised by the governing PDE equation, its training is unsupervised without requiring PDE solutions available (8 30).
However, incorporating governing PDE equations in the loss function also leads to one of the fundamental limitations in PINNs: it has to be re-trained each time when the configuration of the underlying PDE changes (30 g. There have been increasing interests in addressing this limitation (88
I 16 36). Most recent successes have been built on the meta-learning concept, aiming to learn a meta-model across range of PDE configurations such that it can either directly generate the PINN for a given PDE configuration (.) or can be rapidly fine-tuned to a given PDE configuration
29
16 36 22 Treating PINN-training for each PDE configuration one task, these meta-models are trained across a set of tasks uniformly sampled from a range of PDE configurations.
When training the PINN, however, it is known that the difficulty for solving the underlying PDEs varies. Examples of determinants for such task difficulty include: 1) the value of the PDE parameters
Both authors contributed equally to this work
Source code available attttss:: //g thub. com http:bbccommmaaaaaamaa ou D CLR2024

---

Published as a conference paper at ICLR 2024
A
Easy PDE
Difficult PDE
Easy
Difficult PDE
Figure I: PINN solutions for the Burger equation with low versus high Reynolds numbers, obtained higher budget (A) versus lower budget (B) of residual points for training.
or initial conditions being considered (9). and 2) the available budget of sampling points on which PDE residual loss can be calculated (commonly referred to "residual points") (34). For instance, Burger equations with high Reynolds numbers are associated with PDE solutions with sharp transitions illustrated in Fig. [7A. presenting more significant challenges for PINN to solve (19). As the sampling budget residual points decreases, the difficulty of PINN training increases, although much more significantly for the higher Reynolds numbers (Fig. [B). The existing meta-learning approaches to PINNs which sample uniformly across all PDE configurations each with the same budget of residual points (28) 29 16 36 neglect such differences in task difficulty. This can lead to degraded average performance as well as performance disparity among various PDE configurations.
In this paper, introduce novel difficulty-aware task sampler (DATS) for meta-learning of PINNs, with a goal to prioritize different PINN tasks depending on the task difficulty associated with the underlying PDE configurations. We are motivated by the recently presented concepts in general meta-learning that optimize task sampling probabilities during mela-training to minimize the average performance of all tasks during meta- validation (35). While recent works approach this through neural optimization procedures realized via reinforncement learning (23 (3),, we theoretically establish analytical solutions for the optimal task sampling probability across the range of PDE configurations considered. In the practical setting where discrete set of PDE configurations is considered, we then present a novel strategy to convert this sampling probability to dynamic allocations of residual points among different PDE configurations (DATS-rp), and contrast it with a simpler strategy that directly weighs the contribution of different tasks to the meta-training loss (DATS-w).
We note that there is not yet any existing adaptive task sampling strategy for meta-learning PINNs. While related concepts exist general meta-learning, they are not trivial to directly extend to the PINN due to its unique unsupervised learning nature. In parallel, adaptive sampling strategies being studied in PINN literature but at the level of sampling "residual points" for training a single representative approach is based self-paced learning that automatically schedules the sampling of training residual points based on the difficulty level of obtaining PDE solutions those points (II. As this is relevant the concept of "difficulty-awar" sampling" in DATS, we adopt the methodology of self-paced learning (20) described in (L but extend it (from the original level of sampling residual points) to the level of task sampling in meta-learning PINN: this gives us two primary baselines for evaluating DATS: uniform task sampling routinely used in meta-learning PINNs, and self-paced task sampling as a stronger adaptive task sampling baseline.
PINN 34 27 9 25
To demonstrate that the benefits of DATS task-level sampling is orthogonal to residual-poin sampling strategies used for learning each PINN, we evaluate DATS and its two task-sampling baselines in combination with three representative residual point sampling strategies (34). To further demonstrate that DATS is agnostic to the underlying choices of meta-learning approaches, we conduct experiments on two representative existing meta-PINNs (I 16). We conduct these experiments in four benchmark PDEs commonly used in PINN literature and, to demonstrate the importance of considering task difficulty in meta- meta-PINNs, those commonly used in literature. We further demonstrate that DATS can be used to meta-learn across more general PDE configurations such as varying initial conditions.
consider ranges of PDE parameters and residual- -piint budgets beyond
Throughout all benchmark PDEs, meta-PINN frameworks, and residual-poing sampling strategies considered, DATS demonstrated its ability to significantly improve the average performance of all PINN tasks while reducing their performance disparity, all achieved at substantially smaller budgets of residual points in comparison to uniform or self- -aced task- sampling in meta-learning PINNs.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
Meta Learning of PINNs: A PINN is typically trained to solve PDE with a given configuration.
By considering different PDE: configurations as different tasks, there is a recent interest in leveraging meta learning techniques to approximate the solution manifold over the PDE configuration space (28). Recent meta-PINN approaches can be categorized based on whether it leverages feedforward- or agnostic meta-learning (MAML) based meta-learning frameworks. In feedforward-based meta- PINNs, a meta-model is learned to map the PDE configurations to PINN weight parameters (I) This mapping is learned as hypernetwork (12) in HyperPINN (I. while learned with supervision by the data pairs of PDE configurations and PINN weights in (28) Meta-MgNet ( introduces a
hypernetwork within the Multi-grid Network (MgNet). In MAML-based meta-PINNs, an effective initialization for the PINN weight parameters is learned such that, with a few gradient updates, the network can be fine-tuned to a new given PDE configuration (22) 16). This is achieved in (22) by Reptile-based strategy to directly learn the initialization of the main PINN. In MAD-PINN (6), a latent embedding is introduced as an additional input to the main PINN and is fine-tuned to represent an implicit encoding of individual PDE configurations. In (36), MAML is applied to train and fine-tune a meta- Peta- for ID parametric are models. Until now, all these meta-PINN works treat all individual PDE tasks/configuratinn difficulty associated with the PDE configurations and their impact on the meta-model. DATS is the first task-sampling strategy for meta- meta-PINNs and is agnostic to the type of meta-learning approaches used: to demonstrate this, we evaluate DATS on two meta-PINN models representing each of the feedforward- and MAML-based approaches: HyperPINN 11 and MAD-PINN (16).
equally, neglecting the potential differences in task
A recently introduced GPT-PINN 55 solves a similar problem by extending the idea of reduced order model: it learns to use pre-trained PINNs for finite set of PDE configurations activation functions in the PINN of new PDE. To determine when to add a neuron for a given PDE configuration baseline in a subset of experiments.
inherently considers its difficulty. We thus included
Adaptive Residual Sampling in PINN:
For training an individual PINN, the most commonly used
strategies for selecting residual points is uniform random sampling (34) from a uniform distribution or equi-spaced grid. There however has been an increasing interest in adaptive sampling techniques Examples include the residual- resedua--based adaptive refinement (RAR) (5) to sample more residual points in areas with significant PDE losses, and residual- -based adaptive distribution (RAD) that resamples residual points from probability mass function defined over the PDE loss 34. Recently, self-paced learning is also introduced to enable adaptive residual point sampling by automatically scheduling their inclusion in training based on their PDE losses .T..
for residual points 34 g
Despite the shared motivation in difficulty-aware sampling, adaptive residual sampling and DATS are fundamentally different: the latter addresses task-level sampling when meta-learning PINNs, while the former addresses sampling at the level of residual points when learning a single PINN. To understand how the benefits of task sampling may be influenced by the lower-level residual points sampling, we will evaluate DATS and its samp-ign baselines in combination with three residual point sampling strategies: random, random with resampling (random-R) (25), and RAD that was reported as the best performing residual point sampling technique in a recent survey (24)
Adaptive Task Sampling in General Meta Learning: In general, it is increasingly recognized that different tasks may have different learning difficulties with different impact on the meta-model
(4). To address this, a probabilistic active learning method is designed in (18) to rank the latent task embeddings using a utility function. In (2D) a task difficulty measure defined over pairs of classes to define an evolving task selection probability. In 21). tasks are prioritized based on their contribution to the generalization of the meta modal meta- valiaavai,,,,nn, realized via reinforncement learning. None of these task measures or optimization strategies is trivial to extend to meta-PINNs.
DATS shares the motivation of (2 35) but approaches the bi-level optimization with a theoretically backed analytical solution, establishing the first adaptive task sampler for meta-learning PINNs. While there is not an existing baseline nor a trivial extension that can be achieved from general meta- learning, the self-paced adaptive sampling of PINN residual points mentioned earlier (lI shares some high-level similarity to DATS. We thus extend MD originally designed for residual point sampling - to task sampling when meta-learning PINNs. This establishes an adaptive task- task-sampling baseline for DATS, with a key difference that DATS optimizes task sampling to minimize the meta-model's validation loss, whereas self-paced learning does to minimizes the meta-model'ss training loss.
3

---

Published as a conference paper at ICLR 2024
Alternative Deep Learning Approaches to Solving PDEs: There alternative deep learning techniques for solving parametric partial differential equations, such as operator learning (e.g.. DeepONet (24)) that, unlike unsupervised PINN trianing, often relies on supervision from explicit PDE solutions. Emerging works have also attempted to combine these two approaches resulting in physics-informed DeepONets (PIDeepONets) (33). As DATS is designed to address challenges associated with meta-learning unsupervised PINNs, comparison with other supervised PDE-solving solutions is outside the scope of this work.
PRELIMINARIES
Learning PDEs with PINNs:
PINNs approximate the solution u(x) to PDE, by training neural network (x) parameterized by ó to satisfy the given PDE and boundary/niiiall condi-
tions. Assuming a PDE in the general form of F(u, Vu, viu,..,, A), the PINN loss function CPINN(E, us A) = CBI + LRsiduual includes two terms:
CResidual (2r, úg. A) FF (Ug
Vûg: V I. A)P where x IE
(1)
Oūs
(B, uo, A) |u(x) - (1))
s(u) where Oor E 0B
On
where LResidual tied specific value of the PDE parameter X, computes the PDE residual on set of residual points x E P. sampled from the solution domain without knowing the corresponding PDE solutions. LBI defines the boundary/initial conditions As the PDE parameter or boundary/initaal conditions change, the PINN has to be re-trained from scratch, main challenge for training PINNs for parametric PDEs over a range of configuration values or boundary/initial conditions.
Meta-Learning PINNs: Meta-learning has been increasingly applied to PINNs to learn a meta model, with meta-parameters 0, that can quickly adapt a PINN ú 0) to any given PDE configuration À ~~ p(A). We consider two representative meta- PINNs: HyperPINN (I and MAD-PINN (16).
HyperPINN ①I) formulates the meta-model as hyper-network that maps PDE configuration À to the weight parameters & of main PINN as: ø He (A), such that (s(2;) 0) (2)()) The
hyper-network is optimized across a range of PDE configurations with:
f arg min Es~p(A) [C H A)]
PEP,
(3)
MAD-PINN (16), in contrast, provides an additional input zx to regular PINN as (a( 0) ue (Ix 2x). zx, termed as implicil code, vector that is optimized individually for each PDE configuration À along with the weight configuration & of the PINN, over a range of À via:
= (1 x²) = arg min min (EAp(())(CPINNC ús(z, 2x), A) + G² | : Z (4)
0..>x
where 11 ||ax, 1| enforces training stability and a is haper- When a new PDE configuration arises, MAD-PINN achieves fast transfer by fine-tuning ΣΛ (or along with Φ) to the new task.
Therefore, HyperPINN resembles a feed-forward meta-learning approach (5)), whereas MAD-PINN resembles a MAML approach (TO). HyperPINN does not require test-time fine-tuning, although considers meta-tran meta-training only across PDE parameters whereas MAD-PINN can accommodate broader definitions of PDE configurations. They thus provide two diverse settings to test DATS.
METHODOLOGY
In existing meta-PINN works, optimization of equation 3 or equation is done over range of X's sampled from a uniform distribution of p()), with the same number of residual points used in each task. This treats all PDE configurations equally and does not take into account the difficulty levels of solving different PDEs. Instead, we propose DATS to optimize the sampling probability p(A) for meta-training, such that the resulting model will minimize the average validation loss follows:
p" (A) arg min E Ex- LrINN (z, (z( A,))))
J. E Peals
p())
(z; 0*) ={
ūHH (x) 0" solution of
(5)
uz, 2t) 8" = ((2ī), x*) = solution of
4

---

Published as a conference paper at ICLR 2024
Here, CPINN (2,, (z;xx; 8,), X) denotes the loss of the PINN (x;:;0" meta-validation residual points x E Pvol. calculated over uniformly sampled tasks À ~ (A). The individual PINNs üs 8*) optimized from the meta-learning objective (equation 3 equation 4 over the meta-training task sampling probability p()). This introduces a nested bi-ievel optimization where the optimization of p()) in equation 5 is defined over the optimization of equation equation 4 the given p(A).
Because of this nested optimization, existing works (23) 35) have resorted to reinforcement learning
solve equation
To reduce this complexity, we derive analytical solutions to equation S for DATS.
4.1 ANALYTICAL SOLUTIONS
For readability, below we denote the validation loss in equation Luat and the training loss as len, In an iterative optimization scheme to solve equation 5 instead of actually optimizing (a; 1 over p(A) via equation 3 equation at each iteration L, we note that 0++1 can be approximated by single-step of gradient descent @ H 77 .s alee. dA. With this, the validation
loss at iteration 1, beat, 10*+1), can be expressed (00 n.. (ote,, d'A) which, with first-order Taylor expansion, can be expressed as:
Laat,x (00)
Volvat,x(0))
(6)
Substituting equation 6 into equation 5 and letting Volend = Itr.A and Velval, = Ival,ds we have: (2)
arg min Ex-u(() (lval (00) -n pp(ppp(pppppp p(A)ger, (0a))
(7)
p())
arg max magmaxx BA-u p(Noppppllllle p(N))emaa( (0')
P(A)
where the underlined integral represents expectation of the training gradient over p(A), and the following term represents the validation gradient on a particular value of X. This use of first-order 6 and assumption of the single-step gradient descent for the optimization nested within is inspired by the first-order meta-learning algorithms described in Reptile (26). It allows equation 7 to be optimized analytically when iterating with the optimization of the meta- PINNs: at iteration t, task-sampling probability p' (A) is first optimized by equation 7 using the current training and validation loss all task X's; the meta- metal (+++1 1 is then updated given p' (A).
approximation in equation
Regularization Strategies: To stablize task probability assignments, we further regularize p(A) to prior distribution r r(A) with the Kullback-Leve
(KL) divergence and its strength modulated by hyperparameter B. We consider two options for r(X): 1) r(A) U(r) uniform prior (uniform-KL) or 2) r()) = p(X') optimized at the previous iteration (consecutive KL). We will ablate these two regularization strategies in experimental evaluations.
4.2 DISCRETE APPROXIMATIONS
Considering a typical setting of meta-learning PINNs with a discrete set of PDE configurations (2., )n]] we further obtain discrete approximation of the objective function in equation 7
n
p(Ai)gm.)((8))) Iuat. (f')
72 Σ p(>:)u, wn = (9tm., (0'), Σ goal,,, (8')
(8)
T 2=1
j=1
where (.) denotes the dot product. Intuitivtely, w; measures the gradient similarity between the training loss of task X; and the validation loss across all tasks. This results in a higher w; to a PDE configuration A; that is most beneficial for reducing the valudation loss across all tasks. Based on Karush-Kuhn Kuhn-Tucker (KKT) first-order necessary condition for an optimal solution, we can then derive analytical solution to our optimization as:
1 (w))
++1))) = Ti expl w/)) Σ ri exp( ws)
where r; = 1 for uniform-KI regularization, and ry = p'(A,) for consecutive-Ke KL regularization (derivations are included in Appendix A). Considering a total budget of by residual points and task- specific budget of bx., we now introduce two strategies to utilize p (A,) to realize the Meta-PINN training objective in equation 3 equation over the discrete representations of X.
5

---

Published as a conference paper at ICLR 2024
DATS-w - Adaptive Weighting of Meta-Training Losses: As (X,) determines the sampling probability for PDE configuration A, the most intuitive solution based on importance weighting would be to interpret p' (A²) as adaptive weighting to control the contribution of each PDE task to the training loss. The meta-training objectives in equation 3 or equation 4 will then become:
by/n Σ (0, k=1
0++1 aarg min bt
JA, E (P.rl br/n (10)
where Paur denotes the training residual points for PDE configuration A; and Px,_l denotes size: in another word, the budget of residual points is assigned equally across all tasks as in existing works.
DATS-rP- - Adaptive Allocation of Residual Points: Considering the importance of residual points for PINN training, we consider a novel alternative that utilizes paaeesspp///// (Ax) to control the allocation of resources for each PDE task. Instead of assigning residual points uniformly across all tasks, we will adaptively change the budget of residual point sampling for different tasks via Px.r = =p+ (X,) bby, such that more residual points will be assigned to tasks receiving higher p (As). This results in:
(da)abr
g++1 arg min br
Σ
Itr, (x, TAi,k E PAI,T IA.. = p by (11)
In experimental evaluations, we will carry out ablation studies to consider the efficacy of the adaptive weighting vs. resource allocation strategies above, along with the two KL-regularization strategies.
EXPERIMENTS
We evaluated DATS in both HyperPINN ( and MAD-PINN (16), in comparison with two baselines of task sampling strategies: 1) uniform task sampling as used in all existing works, and 2) self- paced task sampling. Furthermore, on the Burger equation, we considered the additional baseline of GPT-PINN (5). We focused on three categories of evaluations. First, on the Burgers' equation (3) - one of the most commonly used benchmarks in PINNs - we conducted ablation studies to understand the adaptive weighting vs. residual-poin allocation strategies, the uniform consecutive- KL regularization strategies, and the effect of the hyperparameter 3. Second, we carried out an extensive set of comparison studies among DATS, uniform, and self-paced task sampling in the presence of three residual point sampling strategies: random (30 (34), random-R (25), and RAD as described earlier (34). Finally, on five benchmark PDE equations: Burgers' 3 convection 11 14).
one
sraction-diffusion (19 14, 2D Helmholtz (31) equations, and 3D Navier-Stokes equation (17) we performed comprehensive evaluations on DATS vs. its baselines across different residual-poii sampling budgets. More specifically, we set up the upper and lower bound of the meta-PINN PDE solution errors for the baselines using, respectively, small and large residual point budget. We then evaluated DATS with a series of budgets in between, testing the hypotheses that 1) DATS will achieve a performance improvements compared to its two baselines at a given budget, and this improvement will be more significant at smaller budgets; and 2) DATS will need much smaller budget to achieve the same performance of its two baselines. We measured the performance of the meta-PINN, with: 1) average L2 error of the PDE solutions, and 2) performance disparity among the PDE configurations considered, defined as the performance difference between worst-performing to best pest-performing PINNs. In addition to computing these metrics for PDE configurations included in the meta-training, which is the most common way PINNs are evaluated, we further examined these metrics when HyperPINN was generalized to new PDE configurations not included in the meta-training. We left MAD-PINN out of the latter test as it relies on fine-tuning to generalize which we expect to confound and also reduce the benefits of DATS. More implementation details are included in Appendix
5.1 DATS ABLATION STUDIES
We ablated the major components of DATS using the ID Burgers' equation. The special characteristic
of this PDE, i.e., the existence of the shock wave, makes it challenging for PINNs to learn, especially for Burgers' equation with large Reynolds number (R) such À 1/R x le - 3. In this experiment, we considered 12 values of À E (1e -ie-33 3, 0.1) in meta meta-training and a total residual budget of by 1000 12. The base model used was HyperPINN with the original architecture introduced in

---

Published as a conference paper at ICLR 2024
1.2 Error
Disparity
KL
3
DATS-
DATS W
DATS rP
DATS-W w
Uniform
10
0.064 + 0.009
0.151=0.056
0,075 = 0.026
0.188+0.133
Uniform
0.205 + 0.075
0.147=0.051
0.173 0.034
0.196#0.041
Uniform
0.1
0.168 = 0.052
0.153+0.087
0.188 + 0.095
0.2072-0.128
Consecutive
10
0.233 # 0.057
0.210=0.022
0.205 = 0.029
0. 0.11620.035
Consecutive
0.129 + 0.049
0.191
0.179 0.075
0.207=0.093
Consecutive 0.1
0.169 # 0.063
0. 0.18320.106 103
0.179 # 0.075
0.177=0.091
Unifrom Sampling
0.248 0.027
0.212 0.043
Table 1: DATS ablation of adaptive weighting versus adaptive residual point allocation, and uniform- KL versus consccutive-Ki regularizations under difference values of the hyperparameter 3.
Unakem
0.0
kondom
Hhdd
ManSon:
1andom
Figure 2: L2 errors (left) and disparity (right) of HyperPINN when using DATS, uniform, and self-paced task sampling in combination with random, random-R R, and RAD residual point sampling.
(. We investigated the effect of 1) uniform-Ki KL versus consecutive-KL regularization, and 2) adaptive weighting versus adaptive residual-poin allocation strategies, with three choices of 3 {0.1, 10). The results are summarized in Table [ All variations of DATS outperformed HyperPINN with uniform task sampling. DATS-rp with uniform KL regularization, using hyperconfiguration
10, achieved the best performance throughout all experiments. Thus the experiments reported in rest of the paper considered DATS-rp with a uniform KL regularization, where ,3 was tuned using grid search for 3 E {0.1, 10} for each PDE equation separately.
5.2 THE EFFECT OF RESIDUAL POINT SAMPLING ON TASK SAMPLING STRATEGIES
We tested how the comparison of DATS and baselines may be affected by the residual point sampling strategies. We continued to use HyperPINN on the Burger's equation for 6 values of X (le 0.1) and total residual budget by = 3000 x 6. We considered three residual point sampling techniques: 1) random, 2) random with resampling (random-R) (25), and 3) RAD (34). As shown in Figure 2 DATS consistently outperformed the uniform and self-paced baselines, regardless of the residual point sampling strategies (except in the case of RAD, where all three methods obtained comparable disparity metrics). These showed that the effect of adaptive sampling at the task level is more significant than and orthogonal to the effect of adaptive sampling the level of residual points. Interestingly, contrary to the recent findings on the benefit of adaptive residual point sampling (34), in our experiments random residual sampling was more effective. This suggested that effective residual sampling needed for meta- meta-learning PINNs may be different from that needed for learning a single PINN, revealing a gap of knowledge for further research for the PINN community.
5.3 DATS, UNIFORM, SELF-PACED AND GPT-PINN UNDER VARYING BUDGETS
Finally, we investigated the performance of DATS versus its uniform and self-paced task sampling baselines under varying budgets of residual points, for both MAD-PINN (16) and HyperPINN !I) across five benchmark PDE equations. For Burger equation, we also considered the additional baseline of GPT-PINN. For each equation, we considered wide range of their PDE parameters informed by literature. To further demonstrate that DATS is not limited to meta- mecaallarnnnng across PDE parameters only, we also considered meta-learning across different initial conditions u 0) = a aisin(an (si((azz) for the convection equation considering MAD-PINN, HyperPINN is included in this sub-study it was only designed to handle varying PDE parameters in its original work. Table summarizes the range and number of PDE configurations considered in each equation.
Results on Solving PDEs: Figure summarizes the results for varying PDE parameters across the first four equations and two meta-PINN models. The r-axis denotes the maximum residual point
7

---

Published as a conference paper at ICLR 2024
PDE
Configuration
#Training
#Test
Burger Convection
Ur uux Augz 0
A (0.001,
14
6
Ut + 3uu, 0:
3 (0) (a1, 02) E (1.3)
5
sin (222) Ue Vuge au( u)
uxx
9
Reaction Diffusion Helmholtz (2D)
V, a (1,5) 5)
9
Uvyy + k2 Un.y = q(a, b, b, x.
a1 02 (0.5, 1.5) 9
Table 2: The range and number of PDE configurations considered in each PDE benchmark.
Burger
Convection
C) Reaction Diffuston
Helmholtz
HyperPINN
Dapanty
LO44
MAD
Dicarty
Uniform Self-Pacce
GPT-PINN
DATS
HyperPINN Generalization L3 Loss
100
Figure 3: L2 errors and disparity metrics on HyperPINN and MAD-PIN comparing DATS at different residual budgets with baselines obtained the lower and higher ends of the budget spectrum.
budget per task, and the higher- and lower-bound of PINN errors obtained by the baselines were established at the two ends the budget spectrum. All results consistently confirmed our hypotheses. First, the same budgets (the ends of the budget spectrums), DATS consistently out-performed all the baselines, including GPT-PINN on Burger, with a more significant margin of improvements at the lower end of the budgets. Second, DATS was able reach the lower bound of both the L2 errors and disparity measures of the two baselines (obtained using budget 10,000 residual points per task) using only a fraction of their budgets, i.e., ranging from 40% HyperPINN on the Burgers' equation, to less than 10% Helmholtz, and to less than 1% on the convection equation comparing uniform baseline's budget. A similar trend is observed the results varying initial conditions on the convection equation, as summarized in Figure D.7 in Appendix D.7
Figure 4A provides examples DATS and GPT-PINN solutions: while GPT-PINN's ability to recognize harder tasks allows good performance on harder Burger parameters (e.g., X 0.001),
this seemed to be at the expense of the performance on easier Burger parameters (e.g., X 0.1). In comparison, DATS was able to balance its performance across difficult and easy tasks. Figure 5 providees additional visual examples of the PDE solutions obtained with uniform and DATS task sampling selected residual budgets. More examples can be found in AppendixD.66
8

---

Published as a conference paper at ICLR 2024
A] Burger
B) Navier-Stokes 3D
0.5
t=
1+1
Raconstruction
Generalization
Figure 4: A) Examples of PDE solutions obtained by GPT-PINN and DATS on the Burger equation. B) L2 error and disparity metrics of DATS vs. uniform task sampling on 3D Navier-Stokes equation.
(Budgeb-200)
(Budget=500)
Boxsel Recendraterr
DATS
Doxad Bock
Rocombrudte
Croand Trast
Racomstsction
Gread trut
1
1+0
1=0
1.0.5
1
1.0.5
Figure 5: Visual examples of PDE solutions obtained by DATS and uniform task sampling on the convection (left) and reaction diffusion equation (right).
Figure 4 summarizes the results DATS vs. uniform task sampling on HyperPINN on the more complex 3D Navier-Stokes equation (17) with the PDE parameter
[0.5, 1.5]. Similarly, for PDE configurations included in meta-training, DATS was able to use 40 - 60% of the budget to achieve the performance uniform task sampling achieved using budget of 10,000 residual points per task, and only 20 40% of the budget for generalizing to PDE configurations unseen in meta-training. More details about this experimental setup are included in Appendix C
Results on Generalizing to New PDEs: The third row Figure T shows L2 errors in PDE solutions when HyperPINN applied parameters outside the meta-training sets. The trend is similar that observed in meta-trained PDE parameters, with notable improvements in some equations like Helmholtz. Detailed results on the disparity metric are Appendix D.3
Computational Cost: In practice, DATS update of p" (A:) in equation 18 can be computed every few iterations: through experiments we found that a period of 200 could potentially outperform period of while being more computationally efficient (more details presented in Appendix D). With this. the computational overhead by DATS was negligible: on the same computing environments, in HyperPINN, DATS was 3% slower than uniform and 1% slower than self-paced baselines; in MAD-PINN, DATS was 8% and 6% slower than uniform and self-paced baselines, respectively.
LIMITATIONS AND FUTURE WORK
In this paper, introduce DATS as the first adaptive task sampler for meta-learning PINNs. By optimizing a training task sampling probability that minimizes the meta-model's validation perfor- mance, DATS prevents using unnecessary resources (residual points) for learning casier tasks while improving the performance in learning more difficult PDE configurations. While deriving analytical solutions for the discrete setting commonly used in meta- DeNa-, future works need theoretically examine the convergence of the algorithms presented as well investigate the ability of DATS to optimize over the continuous distribution of Furthermore, the evaluation of DATS can be conducted on a broader variety of more complex and higher-dimensionall PDEs, meta-PINN models, and PDE configurations. Tuning the regularization hyperparameters DATS could be challenging and deserves further examinations. Finally, this work considers DATS only in the setting of PINNs, although its underlying concept may be generalizable to operator learning which is known to require substantial number of training samples of paired PDE configurations and solutions.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This study supported by the National Key Research and Development Program of China(No: 2020AAA0109502) the Talent Program of Zhejiang Province (No: 202 the NIH NHLBI grant R01HL145590, the NIH National Institute of Nursing Research (NINR) grant ROINRO1830 and NSF OAC-2212548.
REFERENCES
[1] de Avila Belbute-Peres, F., Chen, Y.f., Sha, F:: Hyperpinn: Learning parameterized differential equations with physics-informed hypernetwrrks. In: The symbiosis of deep learning and differential equations (2021)
[2] Aziz, A.K.: The mathematical foundations of the finite element method with applications to partial differential equations. Academic Press (2014)
[3] Basdevant, C., Deville, M., Haldenwang, P., Lacroix, J., Ouazzani, J., Peyret, R., Orlandi, P., Patera, A.: Spectral and finite difference solutions of the burgers equation. Computers & fluids 14(1), 23-41 (1986)
[4] Basu, S., Stanley, M., Bronskill, J.F., Feizi, S., Massiceti, D.: Hard-meta-datasett++ Towards un- derstanding few-shot performance on difficult tasks. In: The Eleventh International Conference on Learning Representations (2022)
[5] Chen, Y., Koohy, S.: Gpt-pinn: Generative pre-trained physics-informed neural networks toward non-intrusive meta-learning of parametric pdes. Finite Elements in Analysis and Design 228, 104047 (2024)
[6] Chen, Y., Dong, B., Xu, J.: Meta-mgnet: Meta multigrid networks for solving param- eterized partial differential equations. Journal of Computational Physics 455 2022). http:///oi.org/100 jop.
[7] Chen, Y.. Dong, B., Xu, J.: Meta-mgnet: Meta multigrid networks for solving parameterized partial differential equations. Journal of computational physics 455, 110996 (2022)
[8] Cuomo, S., Di Cola, V.S., Giampaolo, F., Rozza, G., Raissi, M., Piccialli, F: Scientific machine learning through physics-infomin neural networks: Where we are and what's next. Journal of Scientific Computing 92(3), 88 (2022)
[9] Daw, A., Bu, J., Wang, S., Perdikaris, P., Karpatne, A.: Rethinking the importance of sampling in physics-informed neural networks. arXiv preprint arXiv:2207.0233. (2022)
[10] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks (2017)
[11] Gu, Y., Yang, H., Zhou, C.: Selectnet: Self-paced learning for high-dimensional partial differen- tial equations. Journal of Computational Physics 441, 110444 (2021)
[12] Ha, D., Dai, A.M., Le, Q.V.: Hypernetworks. In: International Conference on Learning Representations (2016)
[13] Hao, Z., Liu, S., Zhang, Y., Ying, C., Feng, Y., Su, H., Zhu, J.: Physics-informed machine
learning: A survey on problems, methods and applications. arXiv preprint XXXx:2211100066 (2022)
[14] Hao, Z., Liu, S., Zhang, Y., Ying, C., Feng. Y., Su, H., Zhu, J.: Physics-informed machine learning: A survey on problems, methods and applications (11 2022), htttp://arxiv.orr)
abs/2211 08064
[15] Hospedales, T., Antoniou, A., Micaelli, P., Storkey, A.: Meta-learnig in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 5149-5169 (9 2022). https://doi.org/10..11 109/TAMM /000///M..202111000000000
20212209
10

---

Published as a conference paper at ICLR 2024
[16] Huang, X., Ye, Z., Liu, H., Ji, S., Wang, Z., Yang, K., Li, Y., Wang. M., Chu, H., Yu, F., et al.: Meta-auto-decoder for solving parametric partial differential equations. Advances in Neural Information Processing Systems 35, 13426-23444 (2022)
[17] Jin, X., Cai, S., Li, H., Karniadakis, G.E.: Nsfnets (navier-stokes flow nets): neural networks for the incompressible navier-stokes equations. Journal of Computational Physics 426, 109951 (2021)
(18) Kaddour, J., Sæmundsson, S., et al.: Probabilistic active meta-learning. Advances in Neural Information Processing Systems 33, 20813-20822 (2020)
[19] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., Mahoney, M.W.: Characterizing possible failure modes in physics-informed neural networks. Advances Neural Information Processing Systems 34, 26548-26560 (2021)
[20] Kumar, M., Packer, B., Koller, D.: Self-paced learning for latent variable models. Advances in neural information processing systems 23 (2010)
[21] Liu, C., Wang, Z., Sahoo, D., Fang. Y., Zhang, K., Hoi, S.C.: Adaptive task sampling for meta-learning. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16. pp. 752-769, Springer (2020)
[22] Liu, X., Zhang, X., Peng, W., Zhou, W., Yao, W.: novel meta-learning initialization method for physics-informed neural networks. Neural Computing and Applications 34(17), 14511-14525 (2022)
[23] Liu, Z., Wei, P., Jiang, J., Cao, W., Bian, J., Chang, Y:: Mesa: boost ensemble imbalanced learning with meta-sampler. Advances in neural information processing systems 33, 14463- 14474 (2020)
[24] Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E.: Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence 3(3), 218-229 (2021)
[25] Lu, L., Meng, X., Mao, Z., Kariiadakis, G.E.: Deepxde: A deep learning library for solving differential equations. SIAM review 63(1), 208-228888882202222 228 (2021)
[26] Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. arXiv preprint MxXiv:1803.0299: (2018)
[27] Pang, G., Lu, L., Kamiadakis, G.E.: fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing 41(4), A2603-A2626 (2019)
[28) Penwarden, M., Zhe, S., Narayan, A., Kirby, R.M.: Physics-informed neural networks (pinns) for parameterized pdes: metalearning approach. arXiv preprint 10. (2021)
[29] Qin, T., Beatson, A., Oktay, D., McGreivy, N., Adams, R.P.: Meta-pde: Learning to solve pdes quickly without a mesh. arXiv preprint arX1v:2211.01604 (2022)
[30] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686-707 (2 2019).
[31] Wang, S., Teng, Y., Perdikaris, P.: Understanding and mitigating gradient flow pathologies in physics-ifformed neural networks. SIAM Journal on Scientific Computing 43(5), A3055-A308 (2021)
[32] Wang, S., Wang, H., Perdikaris, P:: Learning the solution operator of parametric partial differential equations with cabi8605 (2021). https://doii.ogg////// 1226//ciav..abi66000 htt tht: science.org. Vol.aabs/10.116// abi 8605
deeponets. Science Advances 7(40),

---

Published as a conference paper at ICLR 2024
[33] Wang, S., Wang. H., Perdikaris, P.: Learning the solution operator of parametric partial differen- tial equations with physics-infomee informed deeponets. Science advances 7(40), eabi8605 (2021)
[34] Wu, C., Zhu, M., Tan, Q. Kartha, Y., Lu, L.: A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering 403, 115671 (2023)
[35] Yao, H., Wang, Y., Wei, Y., Zhao, P., Mahdavi, M., Lian, D., Finn, C.: Meta-leaming with an adaptive task scheduler. Advances in Neural Information Processing Systems 34, 7497-7509 (2021)
[36] Zhong, L., Wu, B., Wang. Y.: Accelerating simulation by meta learning. Journal of Physics D: Applied Physics 56(7), 074006 (2023)
neural network based 1d
A DATS DISCRETE APPROXIMATION
Below we derive Equation (13) from Equation (12) of the main paper using (KKT) first-order necessary condition for an optimal solution. Note again the objective function in Equation (12) in the main paper is:
((p(A.) E -BKLAAA)ssBKKKK/////aa (KI()())))())),
= (gtr,)e Ej=1 Iuat, (0")
(12)
Now, differentiating Equation equation derive:
12 with respect to p(A;) and setting the derivative to zero we
i
a
Σ
p p(Ar) BINn-KKLPPP (KL(D())]
(13)
Op(Ai)
p(As)
w/ - B Blog
(14)
72
r(Ai)
p()p) 1
log
wi
(15)
r(A;) nB
p(A:)
exp(
w.
(16)
r(A,)
n1
Re-arranging Equations (4) for deriving p(Ax) and substituting 3 n2 derive at:
p(A:) r(x)) expl w.)
(17)
Finally, normalizing p(A:) to form the probabilities and replacing r(Ax) with r; we derive at Equation (13).
1 w.)// g Ere exp( wx)) p²((a) = * expl
(18)
where r; can be used for uniform KL regularization, and r, p(),)* for consecutive KL regularization.
B IMPLEMENTATION DETAILS
In this section, discuss the specific hyper-paramen each experiment across all models. Experiments were run on NVIDIA Tesla T4s with GB memory. More information and details can be seen in the provided implementation.
12

---

Published as a conference paper at ICLR 2024
B.1 BURGERS' EQUATION
HyperPINN
PINN
-Fully Connected Layers
Number of Layers:
Hidden layers dimenstion:
Hypernet
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion:
Optimizer: ADAM
Learning rate: le-4 with cosine annealing
Epochs: 20000
MAD-PINN
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 128
Latent Dimension: 16
Latent Loss Regularizer: 0.001
Optimizer: ADAM
Learning rate: 1e-3 with cosine annealing
Epochs: 20000
B.2
CONVECTION EQUATION
HyperPINN
PINN
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 16
Hypernet
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 32
Optimizer: ADAM
Learning rate: 0.001
Epochs: 10000
MAD-PINN
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 16
Latent Dimension: 8
Latent Loss Regularizer: 0.001
Optimizer: Adam
Learning rate: 0.001
Epochs: 10000
13

---

Published as a conference paper at ICLR 2024
B.3 REACTION DIFFUSION EQUATION
HyperPINN
PINN
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 16
Hypernet
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 32
Optimizer: Adam
Learning rate: 0.001
Epochs: 10000
MAD-PINN
Fully Connected Layers
Number of Layers: 5
Hidden layers dimenstion: 16
Latent Dimension: 8
Latent Loss Regularizer: 0.001
Optimizer: Adam
Learning rate: 0.001
Epochs: 10000
B.4 HELMHOLTZ (2D) EQUATION
HyperPINN
PINN
Fuuly Connected Layers
Number of Layers:
Hidden layers dimenstion: 16
Hypernet
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 32
Optimizer: Adam
Learning rate: 0.001
Epochs: 10000
MAD-PINN
Fully Connected Layers
Number of Layers:
Hidden layers dimenstion: 16
Latent Dimension: 8
Latent Loss Regularizer: 0.001
Optimizer: Adam
Learning rate: 0.001
Epochs: 10000
14

---

Published as a conference paper at ICLR 2024
L2
Disparity
Period
DATS
Self- PacfPPaced
DATS
Self-Paced
0.0527
0.0678
0.1317
0.1648
200
0.0385
0.0699
0.0891
0.2049
2000
0.120
0.0707
0.2170
0.2048
Table D.3: Investigating the effect of the period of updating task probabilities.
C SAVIER-STOKES EQUATION
We tested DATS on incompressible Navier-Stokes flow, specifically the firee-dimensional Beltrami flow that adheres to the unsteady, incompressible three-dimensional Navier- Stoker-Sokks equations outlined below:
du 01 + (u Vuju Vpp V in Ω
Re
V 11 0 in Ω
u ur
on Ln
(19)
du 0 I'
u 0) lxx) in 52
where is the non-dimensional time, u(x, () (u, v, w 2 is the non-dimensional velocity vector, p is the non-dimensional pressure, Re is the Reynolds number, 11(x)n is the initial condition and r' and I denotes the Dirichlet and Neumann boundary conditions. respectively. In Beltrami flow, Re=1 and there is the following analytic solution:
2x, -ale" sin(ay + dz) + c cos(ar + dy)le dft v(x, (,, () - aieeysin(ass + dx) ehTh cos(ay + dz) -dft
w(x, 2, alca=sin((e + dy) e TT coss(z + dr) dft
(,t,), = ae2220z + 20y + 2ux + 2sin(ar + dy)cos(az + dy)e
1
(20)
2sin(ay dz) cos ar - dy) (2++2)
2sin(a= dz cos))os(()))))) (ay + dz) dz)e 2d2t
In our experiment, we consider x = Ir, y. 2| E [-1, [-1,1]
[o, 1] and d=l and [-1, [-1,1 x
select the PDE parameter in the range 00.5, 1.5].
D ADDITIONAL EXPERIMENTS
D.1 EYPER-PARAMETER
To decide the period of updating DATS, we conducted grid search for period e {1, 200, 2000]. For fair comparison with self- self-paced learning baseline, also investigated the effect of period of updates on the performance of this baseline as well. As shown in table The results indicate that models period=200 for DATS and period=1 for self-paced learning baseline, were top performing. Note that this experiment is conducted on Burgers equation considering six values of X E (le - 3, le - 1) different than the subset of PDE parameters used for the experiments of the main paper.
D.2 FULL RESULTS ACROSS ALL PDES
Table, D.4 and D.5 explliin the full set of results from figure of section 5.2 on HyperPINN and Mad-PINN
15

---

Published as a conference paper at ICLR 2024
HyperPINN
Burger
Convection
Method
Budget
1.2
Disparity
Budget
1.2
Disparity
Uni.
500
0.1841 0.071
0.296:10.10
100
1.107
0.0177
0.064 = 0.048
S-P
500
0.130 + 0.040
0.308+0.123
100
0.085 = 0.018
0.063 - 0.012
GPT
500
0.301 + 0.063
0.405 + 0.210
DATS
500
0.0791 0.017
0.209
0.0180
100
0.057 +0.002
0.100 = 0.011
DATS
2000
0.068 1 0.006
0.101
500
0.061
100012
0.096 1 0.026
DATS
4000
0.057 + 0.012
0.186=000022
DATS
10000
0.039 + 0.010
1.132--0.039
2000
0.056 0.021
0.061 + 0.047
GPT
10000
0.097 + 0.008
0.193 0.087
S-P
10000
0.052 + 0.018
0.156110.05
2000
0.059 = 0.016
0.061 = 0.024
Uni
10000
0.058 + 0.011
0.179+0.039
2000
0.064 + 0.008
0.082 + 0.029
Reaction-Diffusion
Helmholtz (2D)
Method
Budget
1.2
Disparity
Budget
1.2
Disparity
Uni
20
0.410 + 0.088
0.131 +00060
50
0.298 + 0.056
0.624 +0.1 10
S-P
0.642 + 0.103
0.153
0.043
50
0.0169
0.005
0.017 = 0.007
DATS
20
0.052 + 0.015
0.025
+0.014
50
0.0108 + 0.001
0.017 + 0.007
DATS
100
0.037 + 0.008
0,0180
=0.007
500
0.0086 = 0.001
0.008 = 0.003
DATS
2000
0.028 + 0.012
0.011
0.005
2000
0,009 +0.003
0.008 + 0.005
S-P
2000
0.062 + 0.024
0.029
#0.01
2000
0.0104
0.002
0.013 = 0.003
Uni.
2000
0.063 1 0.029
0.029
10.01
2000
0.009 + 0.002
0.009 = 0.002
Table D.4: DATS vs. Uniform (Uni.) vs. Self-Pace (S-P) vs. GPT-PINN (GPT) from figure 3 of section 5.2 on HyperPINN. For readability, the double horizontal line in the Table divides the group of methods using the same budget of residual sampling points. Within the group the lowest and highest budgets, the best results are bolded among DATS and its baselines. Within the group of DATS with varying intermediate budgets, we underscore the performance better than the best baseline performance achieved at the highest budget.
MAD-PINN
Burger
Convection
Method
Budget
L2
Disparity
Budget
L2
Disparity
Uni.
1000
0.1474 0.066
0.535:10.1 186
100
1.284
0.089
0.053
0.088
S-P
1000
0,23441 0.110
0.356110.191
100
0.376 + 0.084
0.665 1 0.099
DATS
1000
0.111: 0.025
0.429 = 0.102
100
0.167 = 0.032
0.361 = 0.088
DATS
2000
0.113 1 0.031
0.53110.002
500
0.155 1 0.067
0.308 1 0.165
DATS
4000
0.042 + 0.009
0.290:10.114
DATS
8000
0.013 1 0,002
0.11411..14.
DATS
10000
0,010 1: 0,007
0.077-0,053
2000
0.105 10.009
0.241 1,000
S-P
10000
0.015 = 0.009
0.119-00.80
2000
0.121 = 0.036
0.264:10.086
Uni.
10000
0.016 + 0.004
0.13500.036
2000
0.290 + 0.013
0.602-1 0.041
Reaction-
Helmholtz (2D)
Method
Budget
L.2
Disparity
Budget
L2
Disparity
Uni.
10
0.931 + 0.004
0.088 + 0.025
50
0.419-10.027
1.193-10.067
S-P
10
0.953 + 0.067
0.080 = 0.051
50
0.109::0.052
0.039:10.009
DATS
0.051 + 0.007
0.071 0.011
0.027:10.011
0.040-10.018
DATS
0.045 + 0.020
0.058 + 0.008
500
0.027+00.06
0.040:L0.014
DATS
1000
0.035 + 0.002
0.049 + 0.004
2000
0.020
0.022-10.011
S-P
1000
0.053 + 0.019
0.053
0.023
2000
0.025+00.004
0.028+10011
Uni.
1000
0.040 = 0.012
0.061
0.018
2000
0.026
0.033:1.011
Table D.5: DATS vs. Uniform (Uni.) vs. Self-Pace (S-P) vs. GPT-PINN (GPT) from figure of
section 5.2 on MAD-PINN.
D.3 RESULTS ON GENERALIZING TO NEW PDES
16

---

Published as a conference paper at ICLR 2024
MlyperPNNy Generalization
Burger
Convection
Method
Budget
1,2
Disparity
Budget
1.2
Disparity
Uni.
500
0.167 + 0.065
0.025
0.094
0.907 + 0.047
0.080 = 0.063
S-P
500
0.180 + 0.092
0.278 = 0.098
100
0.077 + 0.016
0.034 = 0.012
DATS
500
0.102 + 0.014
0.160 + 0.009
100
0.053 + 0.006
0.022 + 0.012
DATS
2000
0.089 1 0.003
0.135
0.020
500
0.069 1 0.028
0.034 1 0.004
DATS
0.077 1 0.014
0.137
0.026
DATS
10000
0.051 + 0.016
0.084 + 0.031
2000
0.061 + 0.018
0.022 = 0.004
S-P
10000
0.085 + 0.024
0.129 + 0.051
2000
0.067 + 0.015
0.049 + 0.012
Uni.
10000
0.056 + 0.009
0.114 0.016
2000
0.059 + 0.0TO
0.028 + 0.014
Reaction-Diftusin
Helmholtz (2D)
Method
Budget
1.2
Disparity
Budget
1.2
Disparity
Uni.
20
0.357 + 0.133
0.244
0.061
50
0.336
0.081
0.440 = 0.091
S-P
20
0.641 + 0.131
0.504 + 0.448
50
0.087 + 0.059
0.080 + 0.049
DATS
20
0.098 + 0.024
0.095 + 0.018
50
0.096 + 0.062
0.010 + 0.089
DATS
100
0.081 1 0.013
0.102
0.022
500
0.062 1 0.009
0.050 + 0.026
DATS
2000
0.095 + 0.008
0.123
0.015
2000
0.086 + 0.039
0.073 + 0.075
S-P
2000
0.112 + 0.027
0.129 - 0.031
2000
0.057 1 0.020
0.030 = 0.021
Uni
2000
0.105 + 0.018
0.114
0.008
2000
0.098 + 0.018
0.101 + 0.019
Table D.6: DATS vs. Uniform (Uni.) vs Self-Pace (S-P) generalization results for HyperPINN across
all PDEs.
A) Burger
B) Convection
C) Reaction Diffusion
D) Helmholtz (2D)
0.0
014
Budget
Budget
Budget
Budget
Utifern
Figure D.6: HyperPINN generalization results (disparity) on PDE parameters not included in meta- training.
Figure D.6 summarizes the disparity metric when HyperPINN is applied to PDE parameters outside meta- meta-trainin training sets (generalization) and table D.6 includes the full set of results presented the third row of figure.
D.4 RESULTS ON GENERALIZING TO NEW INITIAL CONDITIONS
Figure. D.7 compares DATS with uniform and self-pace baselines on MAD-PINN when tested on initial conditions unseen during training.
D.5 RESULTS ON 3D NAVIER-STORE STOKES EQUATION
Table. D.7 presents the full set of results including L2 and disparity metrics using DATS and uniform lask-sampling baselines for solving 3D Navier-Stokes equation with HyperPINN and parameter configuration [0.5, 1.5). A subset of 6 parameters is used for training and unseen parameters during the training are used for generalization experiments.
D.6 EXAMPLE SOLUTIONS
Loss convergence & examples of visual results: Figures. D.8 showcases validation L2 error comparison of DATS with Uniform and self- learning baselines using HyperPINN during training for the example budget of b = 500 on Burgers' equation. The thirds column in this figure shows how the probabilities assigned each task (pde parameter instance) changes during training. In
17

---

Published as a conference paper at ICLR 2024
Disprity
Budget
Budget
Uniform
Sett- Paced
DKTS
Figure D.7: MAD-PINN results on learning across initial conditions for the convection equation.
3D Navier-Stokes
Reconstruction
Generalization
Method
Budget
L.2
Disparity
L,2
Disparity
Uniform
100
0.048 + 0.008
0.041
0.003
0.041 + 0.007
0.018 + 0.003
DATS
100
0.043 1 0.002
0.028 = 0.003
0.037
+ 0.001
0.014 + 0.001
DATS
TO00
0.036 + 0.002
0.033
0.005
0.032
0.001
0.018 + 0.001
DATS
10000
0.034 + 0.004
0.021 + 0.007
0.031
0.003
0.012 + 0.005
Uniform
10000
0.035 = 0.003
0.03 + 0.005
0.032
0.001
0.017 = 0.002
Table D.7: DATS Uniform task sampling for reconstruction and generalization of 3D Navier-Stokes equation with HyperPINN.
this experiment, DATS probabilities are updated every 200 epochs. Similarly, Figure D.9 compares
MAD-PINN results at the example budget of b 1000. I0.1 D.11 and D.13 visualize example PDE solutions convection, reaction diffusion, Helmholtz (2D) accordingly. Figure D.13 compares MAD-PINN before and after applying DATS on new initial conditions for the convection equation.
18

---

Published as a conference paper at ICLR 2024
PDE solution at A = 1e-3
L2 Convergence
Probability
DATS
Self-Paced
Uniform
Figure D.8: Comparison of PDE solution at le - 3, 1.2 loss convergence, and probabilities at 500 residual points budget using HyperPINN.
solution at 1e-3
L2 Convergence
Probability
DATS
Self-Paced
Uniform
Figure D.9: Comparison of PDE solution 1e 3, L2 loss convergence, and probabilities
1000 residual points budget using MAD-PINN.
19

---

Published as a conference paper at ICLR 2024
HyperPINN Self-Pace
Uniform
DATS
MAD-PINN Self-Pace
Uniform
DATS
Figure D. 10: Comparison of Convection equation solution and 100 residual points budget.
HyperPINN Self-Pace
Uniform
DATS
MAD-PINN
Uniform
Self-Pacc
DATS
Figure 1.11: Comparison of Reaction diffusion equation solution at u, = 2.5, 2.5 and 20 residual points budget.
20

---

Published as a conference paper at ICLR 2024
HyperPINN
Uniform
Self-Pace
DATS
MAD-PINN
DATS
Uniform
Self-Pace
Figure D.12: Comparison Helmholtz equation solution at dj, d2 115,, 1.5 and 50 residual points budget.
Uniform (b=100)
DATS (consecutive KL, b=100)
Less Convergence Paat
Converpence
10-
10-
0A 30
0.
2A.
2002
6030
12000
opech
W
Figure D.13: Comparing MAD-PINN and DATS initial condition u(x; 0) (3rn Top
row indicates L2 loss and bottom row compares the results on 8 = 1 (most difficult task in training).
21


Reviews:
[{"review_id": "NrmwvKMOky", "reviewer": "Reviewer_Z9PQ", "summary": "This paper introduces a difficulty-aware task sampler (DATS) for meta-learning of PINNs. The model takes the variance of the difficulty of solving different PDEs into consideration by optimizing the sampling probability of meta-learning. An analytic approximation of the relationship of meta-model and sampling probability is provided to enhance learning. DATS is shown to improve the overall performance of meta-learning PINNs.", "questions": "1) In Fig.3,4,6,  Why do the uniform and self-paced baselines only have higher and lower bounds of errors, not curves at different residual budgets?\n\n2) How does the DATS compare with the Reduced Basis Method, e.g., [1]?\n\n3) Does DATS also perform well on more difficult PDEs, such as with discontinuity and high-dimension? Examples includes the shock tube of compressible Euler's equation and  2d/3d N-S equation.\n\nRefs:\n[1] Chen, Yanlai, and Shawn Koohy. \"GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs.\" Finite Elements in Analysis and Design 228 (2024): 104047.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Quality: The performance of the approach seems good in the empirical part of the paper and the ablation study is detailed.\n2) Originality: The proposed two strategies to utilize $p^*$ are interesting and the comparison and analysis are comprehensive.", "weaknesses": "The main weakness is the clarity and correctness in both methodology and experiments\n\n1) Lack of intuitive explanation for the math derivations in section 4.1, making it hard to follow.  For example, the $w_i=  \\langle g_{tr}, g_{val} \\rangle$ in Eq.9 may be intuitively interpreted as \"assign the weight according to gradient similarity between train and valid\", I guess.\n\n2) There are some misleading typos and unexplained assumptions in section 4.1.\n* The LHS of Eq(7) should be $l_{\\text {val }, \\lambda}\\left(\\theta^{t+1}\\right)$, but not $l_{\\text {val }, \\lambda}\\left(\\theta^*\\right)$ written in Line 5 of this paragraph.\n* And similarly, the LHS of Eq(8) should be $p^{t+1}(\\lambda)$ not $p^{*}(\\lambda)$.\n* In Line4-5 of this paragraph, the gradient descent of training loss is defined as $\\theta^{t+1} = \\theta^t-\\eta \\int_\\lambda p(\\lambda) \\nabla_\\theta l_{t r, \\lambda}\\left(\\theta^t\\right) d \\lambda$, which assumes that training loss is defined as in Eq(11), the so-called DATS-w. But  DATS-rp loss in Eq(12) does not follow this assumption, thus the analysis is invalid for it. \n* The authors assume that the proposed iterative scheme for $p^*,\\theta^*$ converges and that adding regularization further stabilizes the convergence, without explanations. Intuitively, the first-order Taylor expansion is used in Eq(7), thus the step size $\\theta$ should be small enough to stabilize it. Additionally, the discrete approximations may also introduce errors.\n\n3) The experimental performance of sampling strategies (Section 5.2 and Appendix C) is not reported clearly. There are only figures in the main text. Fig.3,4,6 are difficult to extract information from since the lines and shades overlap heavily.  And since there are no digital numbers available, it is hard to compare the results. For example, in Fig.C.12, it seems Self-pace has the same performance as DATs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWjTDSSJYH", "reviewer": "Reviewer_USEY", "summary": "The paper introduces a novel approach to Physics-Informed Neural Networks (PINNs) with a focus on unsupervised learning for parameterized Partial Differential Equations (PDEs). The authors propose optimizing the probability distribution of task samples to minimize error during meta-validation. They transform the problem theoretically into a discretized form suitable for optimization and introduce two optimization strategies: optimizing the residual points sampled and the loss weight across different tasks. The paper presents experiments on several equations, showcasing improvements over baseline methods.", "questions": "1. Clarify Mischaracterizations: The authors should revisit Section 3 to correct the mischaracterization of data loss and provide a more accurate description.\n\n2. Improve Figure Formatting: Enhancements should be made to the figures to improve readability and clarity, as this will aid in better conveying the results and contributions of the paper.\n\n3. Include Comparison with State-of-the-Art: Adding comparisons with leading methods in the field will provide a clearer benchmark of the proposed method's performance.\n\n4. Discuss Limitations and Potential Overfitting: A section discussing the limitations of the method and addressing potential overfitting concerns would add depth to the paper and provide a more balanced view of the approach.\n\nWith these enhancements, the paper would offer a more comprehensive and clear presentation of the proposed method, its strengths, and its potential areas for improvement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity and Presentation: The paper is well-written, making the novel method and its distinctions from baselines clear. The method is explained in a step-by-step manner, which is easy to follow.\n2. Innovative Approach: The meta-learning method for PINNs is a fresh take on solving parameterized PDEs, and it is well grounded in theory.\n3. Comprehensive Experiments: The authors conduct experiments on various equations, providing a thorough evaluation of their method.\n4. Effective Visualization: The results are presented in a clear manner, with visualizations that aid in understanding the improvements made.", "weaknesses": "1. Mischaracterization of Data Loss: In Section 3, the paper inaccurately defines data loss as the loss of boundary conditions. This is a mischaracterization, especially for Neumann or Robin boundary conditions, which only penalize normal derivatives rather than resulting in data loss.\n\n2. Formatting and Clarity of Figures: Some figures in the paper could be improved for better clarity and understanding. For instance:\n(1) Figures 3 and 4 would benefit from added grids and key values annotated directly on the figures.\n(2) The scales in Figure 5 are too small to read, making it difficult to interpret the results.\nThe authors should review and adjust these figures to enhance clarity.\n\n3. Lack of Comparison with State-of-the-Art: The paper could be strengthened by including a comparison with state-of-the-art methods in the field, providing a clearer context of the method's performance.\n\n4. Limited Discussion on Limitations: The paper does not adequately discuss the limitations of the proposed method, which is crucial for readers to understand the potential challenges and boundaries of the approach.\n\n5. Potential Overfitting: Given the nature of the meta-learning approach, there could be a risk of overfitting to the tasks at hand. The paper could benefit from a discussion on how this risk is mitigated or how the method performs under such circumstances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tHZ91ps1UU", "reviewer": "Reviewer_LK9J", "summary": "This paper presents a sampling strategy for meta-learning in physics-informed neural networks (PINNs). The key idea is to conduct sampling based on the difficulty. The authors provide an analytical solution to optimize the sampling probability, with a regularization term. Experiments show improved performance over uniform sampling. An ablation study has been presented to help understand the method. The method also shows better performance under the same budget.", "questions": "What is the actual training time improvement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Meta-learning for PINN is a promising solution to generalize PINNs so that we do not have to train from scratch for any new PDE.\n2. The proposed sampling strategy is general and can be combined with the existing meta-learning strategy.\n3. Experiments show the proposed method outperforms the uniform sampling.", "weaknesses": "1. It is necessary to report training costs in terms of running time. Although the method can improve the sampling efficiency, the sampling strategy itself could be more time-consuming than uniform sampling. It is unclear whether the method can bring actual benefits in training time compared with the baseline.\n2. The method is only tested on three benchmarks. Results on more benchmarks are encouraged to test the generalizability of the proposed method, such as Heat, Wave, and Advection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)