Title: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Learning

Parametric PDEs

Meta-Learning

physics-aware
Abstract: Solving parametric partial differential equations (PDEs) presents significant challenges for data-driven methods due to the sensitivity of spatio-temporal dynamics to variations in PDE parameters. Machine learning approaches often struggle to capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety of trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs is essential and that among them, \textit{adaptive conditioning}, allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, a simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully data-driven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, forcing terms and solution domain. *Project page*: https://geps-project.github.io

GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning

Abstract

Solving parametric partial differential equations (PDEs) presents significant challenges for data-driven methods due to the sensitivity of spatio-temporal dynamics to variations in PDE parameters. Machine learning approaches often struggle to capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety of trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs is essential and that among them, adaptive conditioning , allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, a simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully datadriven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, forcing terms and solution domain. Project page : https://geps-project.github.io/

1 Introduction

Solving parametric partial differential equations, i.e. PDE in which certain parameters-such as initial and boundary conditions, coefficients, or forcing terms-can vary, is a crucial task in scientific and engineering disciplines, as it plays a central role in enhancing our ability to model and control systems, quantify uncertainties, and predict future events (Cohen & Devore, 2015). Neural networks (NNs) are increasingly employed as surrogate models for solving PDEs by approximating their solutions (Long et al., 2018; Lu et al., 2021; Li et al., 2021). A primary challenge with these data-driven solvers is their ability to generalize across varying contexts of the physical phenomena. This is especially pronounced for dynamical systems, which can exhibit significantly different behaviors when subject to small changes in PDE parameters.

The usual approach for solving a parametric PDE with neural networks involves sampling instances from the PDE parameter distribution (e.g., the PDE coefficients), and then sampling trajectories for each instance of the underlying PDE (Takamoto et al., 2022). The training set thus consists of multiple trajectories per parameter instance, with the objective of generalizing to new instances. This approach aligns with the classical empirical risk minimization (ERM) framework: it assumes that the training dataset is large enough to represent the distribution of the dynamical system behaviors and that this distribution is i.i.d. (Bodnar et al., 2024). However, given the complexity of dynamical

systems and the variety of PDE terms and parameters, these assumptions are rarely met in practice. To address this difficulty, some methods relax these assumptions. For instance, certain approaches embed the PDE parameters as inputs, aiming to generalize to new dynamics (Brandstetter et al., 2023; Takamoto et al., 2023). While this can alleviate the issue, it does not facilitate generalization beyond the training distribution. Other approaches focus on few-shot settings, where a pre-trained model is fine-tuned for each new context (Subramanian et al., 2023; Herde et al., 2024). As we will see, fine-tuning NN solvers typically requires a substantial amount of data, when it is often scarce in practice. In general, given the complexity of physical phenomena, it is often unrealistic to expect a sample distribution that is sufficiently representative to enable reliable generalization to new instances of the same phenomenon.

Our claim is that neural PDE solvers should be explicitly trained with an adaptive conditioning mechanism to enable generalization to new parameter instances. For practical applications, this mechanism should condition the network on new dynamics using only a small amount of data. We assume access to several environments governed by the same general PDE, each defined by a unique set of parameters, from which trajectories can be sampled. This assumption aligns with approaches in multi-task learning (Yin et al., 2022) and meta-learning (Wang et al., 2022; Kirchmeyer et al., 2022), where network weights are conditioned on a context that encapsulates environment-specific information. Although various adaptive conditioning mechanisms have been proposed (Finn et al., 2017; Kirchmeyer et al., 2022; Wang et al., 2022), they are often limited to specific neural network architectures (Karimi Mahabadi et al., 2021) and struggle to scale effectively with data size or with the number of environments. We then propose an adaptation method that can handle a diversity of physical dynamics, is compatible with various NN backbones, and scales effectively with data size and the number of environments.

After introducing the problem in Section 2, we emphasize in Section 3 the need for adaptive conditioning to solve parametric PDEs and illustrate the limitations of traditional ERM based approaches. We then introduce GEPS, a low-rank 1 st -order gradient-efficient adaptation mechanism for learning parametric PDE solvers. At inference, GEPS adapts to a new unseen environment, by learning a compact context vector c e . This approach is instanciated in two representative settings: (i) purely agnostic approaches, where the solver is trained without any prior physical knowledge, and (ii) physics-aware approaches that combine differentiable numerical solvers with NN components in a hybrid framework. Finally section 5 compares GEPS with alternative adaptation baselines. Our contributions are as follows:

We demonstrate on example PDEs that adaptive conditioning generalizes to multiple and new contexts, while classical ERM approaches fail to handle parametric PDEs.

We propose an effective and scalable adaptive conditioning framework for learning neural solvers. It is based on a first-order meta-learning approach and leverages low-rank adaptation, enabling adaptation to unseen environments in a few shot context. This formulation is versatile enough to encompass both pure data-driven and physics-aware models.

We provide experimental evidence of the performance and versatility of the approach on representative families of parametric PDEs, incorporating changes in initial conditions, PDE coefficients, forcing terms, and domain definitions, thereby covering both in-distribution and out-of-distribution scenarios.

2 Problem Description

We consider parametric time-dependent PDEs and aim to train models that generalize across a wide range of the PDE parameters, including initial conditions, boundary conditions, coefficient parameters, and forcing terms. For a given PDE, an environment e is an instance of the PDE characterized by specific parameter values. We assume that all environments share common global features, such as the general form of the dynamics, while each environment e exhibits some unique and distinct behaviors. A solution u e ( x, t ) of the PDE in environment e satisfies the PDE:

where F e is a function of the solution u e , its spatial derivatives, the PDE coefficients µ and forcing terms f ( x, t ) ; B is the boundary condition (e.g., spatial periodicity, Dirichlet, or Neumann) that must be satisfied at the domain boundary ∂ Ω and u 0 is the initial condition (IC) sampled with a probability measure u 0 ∼ p 0 ( . ) . Environment e is thus defined by a set of parameters ξ = {B , p 0 , µ , f } .

The targeted task is dynamics forecasting, where a neural solver is auto-regressively rolled out on a temporal horizon t ∈ [0 , T ] . The goal is to approximate the evolution operator F e with a neural solver G θ ( u e ( x, t )) parametrized by θ ∈ R d θ , capable of generalizing to various PDE instances ξ , both within (in-distribution) and outside (out-of-distribution) the training parameter distribution.

Therefore, we posit the observation of a set of environments e , each characterized by its specific PDE parameters ξ ; trajectories are sampled for each environment, each characterized by an IC u e 0 . We define E tr as the set of environments used to train our model. In each training environment, N tr trajectories are available D e tr = { u i ( x, t ) } N tr i =1 . For the in-distribution evaluation, the model has already learned conditioning context parameters (described later in section 4.1) for each training environment and is simply tested on new trajectories from the same environments using the appropriate context. For out-of-distribution , the model is evaluated on trajectories from new environments from an evaluation set E ev and is adapted. We then assume access to N ad trajectories D e ev = { u i ( x, t ) } N ad i =1 from the new environments to adapt the network. In our experiments we consider a scarce data, few-shot scenario where N ad = 1 . After adaptation, we test our model on new unseen trajectories from these evaluation environments E ev. This setting is illustrated in Figure 1.

Trainingstage

e; E Etr,with Ntr trajectories per environnent

One-shotadaption

eo

eNe

ei E Eev

Trajectory1

Time

Trajectory

Ntr

Nad = 1

Time

Before introducing our method, we first aim to motivate the need for adaptive conditioning in learning to solve parametric PDEs, while illustrating the limitations of the classical ERM setting.

3 Motivations: ERM versus adaptive approaches for parametric PDEs

The classical ERM approach learns a single model on the data distribution, assuming that the training dataset is large enough to approximate the true data distribution. In practice, data acquisition is often costly and even simple physical systems can demonstrate a large variety of behaviors due to changes in the parameters. This may lead to poor generalization especially with scarce data.

To illustrate this, we will compare the performance and behavior of classical ERM approaches and of our adaptive conditioning method for solving parametric PDEs, on two example datasets: the 2D Gray-Scott and the 1D Burgers equations. In these experiments, we generate for each PDE a series of environments by sampling only the physical coefficients of the PDE ξ = { µ } (more details in Appendix B). We consider in-distribution generalization and out-of-distribution generalization (respectively in sections 3.1 and 3.2, both for an initial value problem (IVP), a common setting where the initial condition u 0 corresponds to the system state at one time t 0 only. We then consider an alternative setting, where the neural solver is conditioned over a sequence of past states instead of

one state only (section 3.3), this is denoted as "temporal conditioning". For all experiments, reported results correspond to the averaged relative L2 loss on 32 unseen trajectories per environment.

3.1 In-distribution generalization for IVP: classical vs. adaptive conditioning approaches

Let us first compare adaptive conditioning and ERM approaches, for in-distribution evaluation, when scaling the number of training environments and trajectories. The models are trained on a range of environments - corresponding to different coefficients of the underlying PDE - and evaluated on the same environments with different initial conditions. Here GEPS is implemented with a classical CNN, while the tests for ERM are performed with four different backbones: the same CNN as used for GEPS but without adaptive conditioning, FNO (Li et al., 2021), MP-PDE (Brandstetter et al., 2023) and Transolver (Wu et al., 2024). Additionally, we also compared with a reference foundation model "Poseidon" (Herde et al., 2024). This model has been pre-trained on a variety of IVP PDE equations and is fine-tuned on our data. Poseidon being trained on 2D data is thus evaluated only for the Gray-Scott equation. For all the baselines, we consider the classical IVP setting where only one initial state is given as input.

Scaling w.r.t. the number of environments. We first examine how the number of training environments affects generalization to unseen trajectories within the same range of environments. The models are trained on 4 and 1024 environments with 4 trajectories per environments corresponding to different initial conditions. The evaluation is performed on 32 new trajectories from the same set of environments. As shown in Figure 2, Transolver, FNO, MP-PDE, and CNN fail to capture the diversity of behaviors and their performance stagnate when increasing the number of training environments. Non-conditioned methods are not able to capture the diversity of behaviors for several environments, regardless of the backbone used, when using only an initial state as input. Poseidon on its side behaves much better on Gray-Scott and is able to capture this diversity of dynamics. Our adaptive conditioning approach (GEPS on the figures) performs significantly better than all the baselines, outperforming also the large Poseidon foundation model. We can also observe that GEPS benefits from being trained on a large amounts of environments, as its generalization performance improves with the number of training environments.

Gray-ScottEquation

BurgersEquation

5e-1

5e-1

Relative Loss (log scale)

Relative Loss (log scale)

le-1

5e-2

le-1

5e-2

1e-2

200

400

600

800

1000

0

200

400

600

800

1000

Number of Environments

Number of Environments

CNN

MPPDE

FNO

?

Transolver

Poseidon

GEPS

Scaling w.r.t. the number of trajectories per training environment. For this second series of experiments, we fix the number of environments at 4 and vary the number of training trajectories per environment from 4 up to 1024. Figure 3 shows the same behavior as for the previous experiments: ERMapproaches rapidly reach a plateau and do not capture the variety of behaviors, even with a large number of trajectories, while GEPS scales well and improves with the number of training samples per environment. We additionally make a comparison with a CNN model trained and evaluated separately for each environment. We plot the average of the models' scores (indicated as "average" on the figure). This is an upper-bound of the performance that could be obtained with ERM models trained from scratch (no pretraining as for Poseidon). Note that this requires as many models as environments and is not scalable. While this performs significantly better than training over all the environments, GEPS matches or surpasses this approach.

Gray-ScottEquation

Burgers Equation

5e-1

5e-1

Relative Loss (log scale)

Relative Loss(log scale)

le-1

5e-2

le-1

5e-2

1e-2

0

200

400

600

800

1000

le-2

0

200

400

600

800

1000

Number of Trajectories

Number of Trajectories

CNN

MPPDE

FNO

>Transolver

Average

Poseidon

GEPS

The experiments highlight that non-conditioned ERM approaches are unable to learn multienvironments datasets for solving IVP, whereas adaptive conditioning approaches like GEPS exhibit strong generalization performance, scaling with the number of training trajectories and environments.

3.2 Out-of-distribution generalization to new environments for IVP: classical vs. adaptive conditioning approaches

Let us now consider the out-of-distribution behavior of the two approaches. The models are trained on a sample of the environments from E tr and their associated trajectories, in the same condition as for section 3.1. They are then evaluated on the trajectories of new environments. We report in figure 4 the out-of-distribution generalization performance of ERM methods and GEPS, for the Gray-Scott and Burgers equations, when pretrained on 4 (left) and 1024 (right) environments, with 4 trajectories per environment. For the test, one considers 4 new environments and evaluate on 32 trajectories per environment. Adaptation (GEPS) or fine tuning (baselines) is performed on one trajectory of a new environment. As for the baselines, we consider CNN and Transolver, plus the Poseidon foundation model for the 2-D Gray-Scott equation only. As above, one may observe a large performance gap between the non adaptive approaches and the adaptive GEPS. This supports our claim on the limitations of pure ERM based approaches to generalize to unseen dynamics configurations and new environments.

Pretrainedon4environments(Gray-Scott)

Pretrainedon1024environments(Gray-Scott)

2×10-1

0.432

4 × 10-1

0.136

Relative L2

3×10-1

0.242

Relative L2

10-1

0.100

2 ×10-1

0.080

6×10-2

10-1

0.096

0.063

4 ×10-2

0.034

6×10-21

3×10-2

GEPS

POSEIDON

CNN

T-SOLVER

GEPS

POSEIDON

T-SOLVER

CNN

100

Pretrainedon4environments(Burgers)

0.781

0.854

2×10-1

Pretrainedon1024environments(Burgers)

0.180

0.136

6×10-1

Relative L2

Relative

10-1

4 × 10-1

3×10-1

0.278

6×10-2

0.054

2 ×10-1

GEPS

CNN

T-SOLVER

GEPS

CNN

T-SOLVER

GEPS

T-SOLVER

CNN

POSEIDON

3.3 In and out-of-distribution generalization performance with temporal conditioning

So far we have considered the classical IVP setting with only one initial state provided at time t 0 . We consider now the situation where the model has access at t 0 to an history of past states and not to a

single initial state only. This is a common setting for learning PDE solvers: this allows the model to infer information on the dynamics and represents a more favorable case for the ERM baselines. We report in table 1 the in-distribution ( In-d ) and out-distribution ( Out-d ) distribution performance of ERM methods and GEPS, for Burgers and Gray-Scott PDEs. We did not consider the Poseidon (Herde et al., 2024) model that has been pre-trained only for one initial state IVPs. For in-distribution results, all methods are trained on 4 environments and evaluated on 32 new trajectories from the same environments. For out-of-distribution, adaptation (GEPS) and fine-tuning (baselines) is done on 1 trajectory per environment; 4 new environments are sampled and evaluation is performed on 32 new trajectories. We consider three different history sizes: 3, 5 and 10. Considering past history helps improve all the models. GEPS and the baselines show close performance for in-distribution, while GEPS is an order of magnitude better than the baselines for out-of-distribution.

4 GEPS method

We introduce our framework for learning to adapt neural PDE solvers to unseen environments. It leverages a 1 st order adaptation rule and low-rank adaptation to a new PDE instance. We consider two settings commonly used for learning the PDE solvers. The first one leverages pure agnostic data-driven approaches, as already considered in section 3. The second one leverages incomplete physics priors and considers hybrid approaches that complement differentiable numerical solvers with deep learning components. The general framework is illustrated in Figure 5.

Adaptation

ALdiag(ce)BL

+

WLi

L(u,ce)

L2(z,ce)

Li(zi-1,ce)

Ln(zn-2,ce)

Ln(zn-1,ce)

+1

Ga

4.1 Adaptation rule

We aim to train a model G θ to forecast dynamical systems coming from multiple environments. We perform adaptation in the parameter-space: some parameters are shared across all the environments while others are environment-specific. Training consists in estimating the shared parameters and learning to condition the model on environment specific parameters. At test time, the shared

parameters are frozen and adaptation is performed on the environment specific parameters only. This setting is common to adaptation based approaches (Zintgraf et al., 2019; Kirchmeyer et al., 2022), however most of them do not scale to large problems while GEPS introduces an efficient scalable adaptation mechanism.

Formulation We adapt the parameters of our model G θ using a low-rank formulation. Most deeplearning architectures can be decomposed into modules or layers - in our experiments we use MLPs, CNNs and FNOs. For simplification let us then consider a layer L i from G θ parameterized by a weight matrix W L i ∈ R d in × d out . Adaptation to an environment is performed through a low-rank matrix ∆ W e L i = A L i diag ( c e ) B L i , where A L i ∈ R d in × r , B L i ∈ R r × d out , c e ∈ R r . The weights of layer L i with the adaptation mechanism are then:

where diag ( c e ) is a diagonal matrix capturing environment specific information and A L i , B L i , W L i are shared parameter matrices across all environments. Ideally, we want c e to capture the number of degrees of variations for our environments. If for example our model G θ is an MLP, the adaptation mechanism for a linear layer L i corresponds to:

where W L i , A L i , B L i , b 1 L i , b 2 L i are the parameters of layer L i , shared across all environments. Only c e is specific to each environment, but shared across all the layers of the network (cf Fig. 5).

Considering a low-rank adaption rule is popular in NLP, where large pre-trained models are adapted to new tasks by learning a low-rank matrix ∆ W = AB added to the frozen weights W of the pre-trained model (Hu et al., 2022). Our approach differs in two ways from this setting: (i) the model is learned from scratch without pretraining, i.e., we learn parameters { W , A , B , c e } jointly, (ii) during adaptation, we can adapt to new environments e ∈ E ev by optimizing only the context vector c e , where it is initialized as c e = ¯ c tr, with ¯ c tr the averaged value of contexts learned during training. We experimentally show in Appendix D.3 that the classical Gaussian parameter initialization proposed in LoRA is inefficient in our context.

4.2 Two-step training procedure

This meta-learning framework operates in two steps: Training and Adaptation at inference. The goal is to learn an initial starting point during the training stage using a sample of environments, allowing adaptation to a new environment by adjusting a small subset of parameters based on a limited data sample from the new environment. Unlike many meta-learning gradient based approaches, GEPS does not involve an inner loop and is a 1 st order method. During the adaptation phase, all the parameters except the context parameters c e are frozen and c e is learned from the new environment sample. This approach ensures rapid adaptation by keeping c e low-dimensional. For simplicity, we refer to parameters shared across environments as θ s and parameters specific to each environment as δθ e ≜ c e , and denote θ e = { θ s , δθ e } . The optimization problem can thus be formulated as follows:

We separate training and adaptation steps into a training and an adaptation loop as described in the pseudo-code Algorithm 1.

4.3 Hybrid formulation for learning dynamics

We consider here an alternative problem to the above agnostic formulation. We assume that part of the physics is modeled through a PDE equation and shall be complemented with a statistical module. This is a common situation in many domains where prior physical knowledge is available, but only in an incomplete form. We follow the formulation in Yin et al. (2021) were starting from a complete PDE, we assume that part of the equation is known and will be modeled with a differentiable solver,

Algorithm 1: Training and adaptation for our method

while it is complemented with a deep learning component for modeling the unknown part. We also assume that the coefficients of the known part of the PDE are unknown and shall be estimated. We thus aim at solving both a direct problem (the NN parameters) and an inverse problem (the PDE coefficients of the known PDE part). We consider dynamics for a given environment of the form:

F e and R e respectively represent the known and unknown physics of environment e and H is a function combining the two components which is unknown in practice. As for G θ , our model of the evolution operator for this physics-aware setting, we will use a simple combination:

where G p encodes the physical knowledge and corresponds to the known part of the PDE physical model and G a is the data-driven model term complementing G p . With this model, we use an autoregressive formulation to generate the full trajectory, using a Neural ODE (Chen et al., 2018) as time-stepper for predicting the state u t + τ as u t + τ = u 0 + ∫ τ t 0 G θ ( u ( τ ))d τ , as illustrated in Figure 15. The model is trained directly from trajectories simulated with the full PDE (known + unknown PDE components) using the MSE loss D e tr (more details in Appendix C):

The NN component is adapted as in section 4.1 through a c e parameter context vector. For estimating the PDE coefficients, we considered two alternatives. One consists in using the learned code c e for adapting the physical parameters θ e p = θ p + W p c e , where θ p , W p are shared parameters across all environments. The second one directly learns the parameters θ e p for each environment by gradient descent on the loss function. In the first approach, only context vectors c e are learned while in the second approach, c e and θ e p are learned jointly during adaptation. The performance of the two methods are similar. For the experiments, we evaluated both and used the better-performing one.

5 Experiments

5.1 Dynamical systems

We performed experiments on four dynamical systems, including one ODE and 3 PDEs. The ODE models the motion of a pendulum, which can be subject to a driving or damping term. We consider a Large Eddy Simulation (LES) version of the Burgers equation, a common equation used in CFD where discontinuities corresponding to shock waves appear (Basdevant et al., 1986). We additionally study two PDEs on a 2D spatial domain: Gray-Scott (Pearson, 1993), a reaction-diffusion system with complex spatio-temporal patterns and a LES version of the Kolmogorov flow, a 2D turbulence

equation for incompressible flows. For the pure data-driven approaches, we make no prior assumption on the underlying physics, while for the physics-aware hybrid modeling problem, we assume that the physical equation is partially known and that the deep learning component targets the modeling of the unknown terms (details on the known/unknown terms for each equation are provided in Appendix B). The setting is the classical IVP formulation when only one initial state u e 0 is provided.

5.2 Evaluation Setting

While in section 3 we compared GEPS with baselines ERM approaches and with a foundation model, our objective here is to assess the performance of GEPS w.r.t. alternative adaptation based approaches. We evaluate the model performance on two key aspects. · In-distribution generalization : the model capability to predict trajectories defined by unseen ICs on all training environments e ∈ E tr, referred as In-d . · Out-of-distribution generalization : the model ability to adapt to a new environment e ∈ E ev by predicting trajectories defined by unseen ICs, referred as Out-d . Each environment e ∈ E is defined by changes in system parameters, forcing terms or domain definition. d p represents the degrees of variations used to define an environment for each PDE equation; d p = 4 for the pendulum equation, d p = 3 for the 1D Burgers, d p = 2 and d p = 3 for the Gray-Scott and the Kolmogorov flow equation respectively (more details in Table 4 in Appendix B). For each dataset, we collect N tr trajectories per training environment. For adaptation, we consider N ad = 1 trajectory per new environment in E ev to infer the context vector c e . Evaluation is performed on 32 new test trajectories per environment. We report in Table 2 the relative MSE: 1 N ∑ N i =1 ∥ y i -ˆ y i ∥ 2 2 ∥ y i ∥ 2 2 .

5.3 Generalization results

Implementation We used a standard MLP for the Pendulum equation, a ConvNet for GS and Burgers equations and FNO for the vorticity equation. All activation functions are Swish functions. We use an Adam optimizer over all datasets. Contrary to Section 3, we perform time-integration with a NeuralODE (Chen, 2018) with a RK4 solver, as it was done for other multi-environments frameworks for physical systems (Yin et al., 2022; Kirchmeyer et al., 2022). Architectures and training details are provided in Appendix E.

Baselines As baselines, for the pure data-driven problem, we consider four families of multienvironment approaches. The first one consists in gradient-based meta-learning methods with CA VIA (Zintgraf et al., 2019) and FOCA (Park et al., 2023). The second one is a multi-task learning method for dynamical systems: LEADS (Yin et al., 2022). The third one is a hyper-network-based metalearning method which is currently SOTA among the adaptation methods, CoDA (Kirchmeyer et al., 2022). As for the hybrid physics-aware problem, we implemented a meta-learning formulation of the hybrid method APHYNITY (Yin et al., 2021), where the adaptation is performed on the physical PDE coefficients only while the NN component is shared across all the environments. GEPS-Phy is our physics-aware model (section 4.3) were all the parameters, PDE coefficients and context vector c e are adapted at inference. We also implemented "Phys-Ad", where we use the same formulation as for the hybrid GEPS-Phy, but adaptation is performed on the coefficients of the physical component only while the neural network component is shared across all environments. All the baselines share the same network architectures than the ones used for GEPS, indicated in the implementation paragraph above. More details on baselines implementation are provided in Appendix E.6.

In-distribution and out-of-distribution results We report results for in and out-of-distribution generalization in table 2 for both the data-driven and the hybrid settings. Across all datasets, our framework performs competitively or better than the baselines for the two settings. Our method is able to correctly adapt to new environments in an efficient manner, updating only context parameters c e . For the agnostic data-driven experiments, GEPS obtains the best results, although being on the same range of performance as other methods.

The main differentiator of GEPS w.r.t. the baselines lies in its lower complexity . In terms of training time, GEPS is way less expensive than gradient-based approaches like CAVIA that involves an outer and inner loop and LEADS, which learns a model specific to each environment. In terms of number of parameters, CoDA needs more training parameters because of its adaptation mechanism relying on a hyper-network. A comparison with the baselines in terms of parameter complexity is provided in table 10. Additional results in Appendix D, show that our data-driven framework adapts faster

than the strong baseline CoDA. Concerning the hybrid learning problem, incorporating physical knowledge leads to better results on all datasets except Burgers, where the fully data-driven method performs better. GEPS-Phy is better on all the datasets but Gray-Scott for which APHYNITY baseline is best. For Kolmogorov, the baselines did not converged at training.

5.4 Scaling to a larger dataset

To further illustrate the benefits of GEPS, we compare it to CoDA, the SOTA adaptation model, on a large dataset with a larger model than the ones used for the previous experiments. We use a dataset generated from a PDE with multiple differentiable terms that encompasses several generic PDEs, inspired from Brandstetter et al. (2022). The PDE writes as (more details in Appendix B):

We report the results in table 3. All the methods are trained using a ResNet (He et al., 2016) architecture, using a context size c e of size 8. For In-d , we trained our model on 1200 environments with 16 training trajectories per environment and evaluated it on 16 new trajectories. For Out-d , we further adapt each model to 10 new environments given 1 context trajectory per environment and then evaluate it on 16 new trajectories per environment.

GEPS outperforms CoDA in terms of performance and number of parameters. This demonstrates the importance of scalable adaptation for few shot learning: GEPS is able to scale on large datasets using deep models while remaining parameter efficient.

6 Discussion and limitations

Limitations We have seen that adaptation is essential for learning to generalize neural PDE solvers, and that within this setting, integrating physical knowledge may help. Adaptation still requires sufficient training samples - both environments and trajectories. These findings are still to be confirmed for more complex dynamics and real world conditions for which the variety of behaviors should be much larger than for the simple dynamics we experimented with.

Conclusion We empirically demonstrated the importance of adaptation for generalizing to new environments and its superiority with respect to ERM strategies. We proposed a new 1 st order and low-rank scalable meta-learning framework for learning to generalize time-continuous neural PDE solvers in a few-shot setting. We also highlighted the benefits of directly embedding PDE solvers as hard constraints in data-driven models when faced with scarce environments.

7 Acknowledgments

Weacknowledge the financial support provided by DL4CLIM (ANR-19-CHIA-0018-01), DEEPNUM (ANR-21-CE23-0017-02), PHLUSIM (ANR-23-CE23-0025-02), and PEPR Sharp (ANR-23-PEIA0008, ANR, FRANCE 2030).

References

Akio Arakawa. Computational design for long-term numerical integration of the equations of fluid motion: Two-dimensional incompressible flow. part i. Journal of Computational Physics , 1(1): 119-143, 1966. ISSN 0021-9991. doi: https://doi.org/10.1016/0021-9991(66)90015-5. URL https://www.sciencedirect.com/science/article/pii/0021999166900155 .

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.

C Basdevant, M Deville, P Haldenwang, J.M Lacroix, J Ouazzani, R Peyret, P Orlandi, and A.T Patera. Spectral and finite difference solutions of the burgers equation. Computers and amp; Fluids , 14(1):23-41, January 1986. ISSN 0045-7930. doi: 10.1016/0045-7930(86)90036-8. URL http://dx.doi.org/10.1016/0045-7930(86)90036-8 .

Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, Jayesh K. Gupta, Kit Tambiratnam, Alex Archibald, Elizabeth Heider, Max Welling, Richard E. Turner, and Paris Perdikaris. Aurora: A foundation model of the atmosphere, 2024. URL https://arxiv.org/ abs/2405.13063 .

Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data augmentation for neural pde solvers. In International Conference on Machine Learning , pp. 2241-2256. PMLR, 2022.

Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. International Conference on Learning Representation , 2023.

Ricky T. Q. Chen. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq .

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork-based meta-learning for low-rank physics-informed neural networks. Neural Information Processing Systems , 2023.

Albert Cohen and Ronald Devore. Approximation of high-dimensional parametric pdes. Acta Numerica , 2015.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2016-December, 2016. ISSN 10636919. doi: 10.1109/CVPR.2016.90.

Maximilian Herde, Bogdan Raoni´ c, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes, 2024.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations , 2022.

Xiang Huang, Zhanhong Ye, Hongsheng Liu, Beiji Shi, Zidong Wang, Kang Yang, Yang Li, Bingya Weng, Min Wang, Haotian Chu, Fan Yu, Bei Hua, Lei Chen, and Bin Dong. Meta-auto-decoder for solving parametric partial differential equations. Neural Information Processing Systems , 2022.

Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pp. 4069-4077. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/kamath21a.html .

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameterefficient multi-task fine-tuning for transformers via shared hypernetworks. In Annual Meeting of the Association for Computational Linguistics , 2021.

Matthieu Kirchmeyer, Yuan Yin, Jérémie Donà, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. International Conference on Machine Learning , 2022.

Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences , 2021.

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research , 24(89):1-97, 2023. URL http://jmlr.org/papers/v24/21-1524.html .

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex), 2021.

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=c8P9NQVtmnO .

Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations, 2023.

Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 3208-3216. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/long18a.html .

Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. Nat Mach Intell , 3:218-229, 10 2021. doi: 10.1038/s42256-021-00302-5. URL http://arxiv.org/ abs/1910.03193http://dx.doi.org/10.1038/s42256-021-00302-5 .

Junyoung Park, Federico Berto, Arec Jamgochian, Mykel Kochenderfer, and Jinkyoo Park. Firstorder context-based adaptation for generalizing to new dynamical systems, 2023. URL https: //openreview.net/forum?id=AW0i0lOhzqJ .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .

John E. Pearson. Complex patterns in a simple system. Science , 1993. doi: 10.1126/science.261. 5118.189.

Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2 2019. ISSN 10902716. doi: 10.1016/j.jcp.2018.10.045.

Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems 37 (NeurIPS 2023) , 2023.

Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems , 35:1596-1611, 2022.

Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. Learning neural pde solvers with parameter-guided channel attention. International Conference on Machine Learning , 2023.

Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems , 2021.

Nilam Tathawadekar, Nguyen Anh Khoa Doan, Camilo F. Silva, and Nils Thuerey. Incomplete to complete multiphysics forecasting - a hybrid approach for learning unknown phenomena. Data-Centric Engineering , 2023.

Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. Neural Information Processing Systems , 2022.

Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances , 2021.

Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries. In International Conference on Machine Learning , 2024.

Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting*. Journal of Statistical Mechanics: Theory and Experiment , 2021, 2021. doi: 10.1088/ 1742-5468/ac3ae5.

Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Neural Information Processing Systems , 2022.

Luisa Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. 36th International Conference on Machine Learning, ICML 2019 , 2019-June, 2019.

A Related Work

We review data-driven methods for learning parametric PDEs and existing works for incorporating physical priors in neural networks, in the context of dynamical systems.

A.1 Learning parametric PDEs

Traditional ML paradigm The majority of current approaches for learning parametric PDEs considers a traditional ERM approach by sampling from the PDE parameter distribution. In MPPDE (Brandstetter et al., 2023), PDE parameters are directly embedded in the graph, allowing generalization to PDE parameters sampled from the same data distribution as those used for training. Neural operators (Kovachki et al., 2023; Takamoto et al., 2023) do similarly but require large training sets and often perform poorly for out-of-distribution (OOD) data. Alternatives to ERM such as invariant risk minimization (IRM) have shown stronger generalization by learning domain invariants (Arjovsky et al., 2020; Krueger et al., 2021). However, even for simple classification problems, IRM methods can fail to capture natural invariances and are extremely sensitive to sampling (Kamath et al., 2021).

Multi-environment paradigm Several works have explored multi-environment/meta-learning settings for dynamical systems. · Data-driven methods like DyAd Wang et al. (2022) adapt dynamics models using a time-invariant context from observed state histories, which are often not accessible. Yin et al. (2022) developed LEADS, a multi-task method for dynamical systems that adapts in the functional space but requires training a new model for each environment. Park et al. (2023) proposed FOCA to address the limitations of second-order optimization of gradient-based meta-learning approaches (Zintgraf et al., 2019). (Kirchmeyer et al., 2022) make use of a hypernetwork for fast adaptation to new dynamics by updating an environment dependent context encoding. While effective, gradient-based and hyper-network-based approaches respectively involve inner-loop updates or parameter increases with respect to context dimension. · Model-based approaches like physics-informed neural networks (PINNs) have been adapted for parametric PDEs (Cho et al., 2023); Huang et al. (2022) used meta-learning with PINNs through context vectors c learned via auto-decoding. One drawback of PINN is their inability to handle scenarios where some physics are unknown.

A.2 Hybrid learning

Data-driven methods with soft constraints Physics losses as proposed by (Raissi et al., 2019) have been used as prior knowledge under the form of soft constraints for neural operators, with the objective to alleviate data scarcity issues (Wang et al., 2021; Li et al., 2023).

Data-driven methods with hard constraints Alternatively, physical priors can be incorporated directly as hard constraints, leveraging the training of neural networks with differentiable physics. It has been particularly successful for accelerating computational fluid dynamics (CFD) by addressing numerical errors inherent in PDE discretization and for augmenting incomplete physics (Kochkov et al., 2021; Yin et al., 2021; Takeishi & Kalousis, 2021; Tathawadekar et al., 2023). Most of these approaches do not consider the parametric PDE setting.

B Dataset details

We present the equations and the data generation settings used for all dynamical systems considered in this work. In table 4, we report the different PDE parameters changed to generate training and adaptation environments.

B.1 Damped and driven pendulum equation

We propose to study the damped and driven pendulum equation. The ODE represents the motion of a pendulum which can be subject to a damping and a forcing term. Even simple dynamical systems such as the pendulum equation can present a variety of complicated behaviors (e.g., under-damped, critically damped, resonance, super/sub-harmonic resonance, etc.), presented in figure 6.

Environment 1

Environment 2

0.75

0

F 0S'0

0

0.25

0.00

70

0.25

0.1

0S'0

00

0.75

I°0

Environment3

Environment 4

0.15

0S0

0.10

0.25

()θ

000

S00

0.25

0.50

0.00

Environment5

Environment6

20

15

10

0

10

0

100

200

300

Time Step

400

500

600

700

008

100

200

300

Time Step

400

500

600

00L

008

The form of the ODE is:

where θ ( t ) is the angle, ω 0 the natural frequency, α the damping coefficient, f ( t ) is a forcing function of the form f ( t ) = F cos( w f t ) , where w f is the forcing frequency and F the amplitude of the forcing. In our case where we suppose we only have incomplete knowledge of the phenomenon, we consider we do not know the damping term. For the initial condition, the pendulum is dropped from a distribution θ ( t 0 ) ∼ U (0 , π/ 12) and dθ dt ( t 0 ) ∼ U (0 , 1) .

Data generation Each environment is defined by changes in the system parameters resulting in different behaviors. In training environments, the pendulum is either subject to damping or forcing, e.g., environment 1 is a damped pendulum ( F = 0 ) while environment 2 is a driven pendulum ( α = 0 ). We generate trajectories using a Runge-Kutta 8 solver. For training, we generated 4 distinct environments, each composed of 8 trajectories on the time horizon [0 , 25] with a time step ∆ t = 0 . 5 . We also generate 32 trajectories on a longer time horizon [0 , 50] per training environment to evaluate in-distribution generalization. For adaptation, we evaluate our method on 4 distinct environments defined by parameters unseen during training. Only one trajectory is generated per environment with a time horizon is [0 , 25] to adapt the model to new dynamics; we evaluate the model's performance on 32 trajectories per environment on a time horizon [0 , 50] . During adaptation, environments are defined such that trajectories are subject both to the forcing and damping term simultaneously.

B.2 Gray-Scott equation

The PDE describes reaction-diffusion system with complex spatiotemporal pattern through the following 2D PDE:

where u, v represent the concentrations of two chemical components in the spatial domain S with periodic boundary conditions. D u , D v denote the diffusion coefficients respectively for u, v and F, k are the reactions parameters. We consider complete knowledge of the physical model, but instead uses a RK4 solver with ∆ t = 1 instead of using an adaptive RK45 solver. We generated environments by changing F, k values, while D u , D v are kept constant across all environments. For experiments in Section 3, we sampled uniformly values for F and k in ranges [0 . 03 , 0 . 04] and [0 . 058 , 0 . 062] respectively. For experiments in Section 5.3, details about the environments are given in table 4.

Data generation We generated trajectories on a temporal grid using an adaptive RK-45 solver. S is a 2D space of dimension 32 × 32 with a spatial resolution of ∆ s = 2 . For training, we generate 4 environments with one trajectory per environment defined on a temporal horizon [0 , 200] . We evaluate in-distribution generalization on 32 trajectories per environment defined on a temporal horizon [0 , 400] . For adaptation, we adapt to 4 new environments in one-shot learning manner. We evaluate the model's performance on 32 new trajectories per environments, defined on a temporal horizon [0 , 400] .

B.3 Burgers equation

Burgers' equation is a nonlinear equation which models fluid dynamics in 1D and features shock formation:

where u is the velocity field and ν is the diffusion coefficient and f ( x, t ) is a forcing term. The unresolved scales R closure (¯ u, u ) and the forcing function is typically unknown, and needs to be directly learned from data. For the experiments done in Section 3, environments have been generated by sampling uniformly ν ∈ [1 e -4 , 0 . 5] . For experiments in Section 5.3, we generated environments by changing the diffusion coefficient or changing the forcing function, as detailed in table 4.

Data generation For the DNS, we generate complex trajectories using a 5 th order central difference scheme using Runge-Kutta 45 solver with a time-step ∆ t = 1e -5 and ∆ x = 2 π 16384 . Such trajectories are particularly costly to generate, therefore, we rather use LES. To obtain the ground truth LES trajectories, we apply a spatial filtering operator on the DNS trajectories. We also down-sample the temporal and spatial grid. Therefore, we obtain LES trajectories with a timestep ∆ t = 1e -3 and ∆ x = 1 256 .

For training, we generate 6 environments with 4 trajectories per environment. Trajectories have been generated on a temporal horizon [0 , 0 . 05] for training and for in-distribution evaluation. For adaptation, we adapt our model on 4 new unseen environments using only trajectory per environment. Weevaluate out-distribution performance on 32 new ICs on a time horizon [0 , 0 . 05] . For the generation of the data, we define a [0 , 2 π ] periodic domain and consider the following initial condition:

Velocity is linked to energy by the following equation :

B.4 Kolmogorov flow

We propose a 2D turbulence equation. We focus on analyzing the dynamics of the vorticity variable. The vorticity, denoted by ω , is a vector field that characterizes the local rotation of fluid elements,

defined as ω = ∇ × u . Like Burgers, we generate LES, leading to an unknown unclosed term appearing in the vorticity equation, expressed as:

Here, u represents the fluid velocity field, ν is the kinematic viscosity with ν = 1 /Re and f ( x, y, t ) is a forcing term. The unresolved scales R closure (¯ u, u ) and the forcing function is typically unknown, and needs to be directly learned from data. For the vorticity equation, environments can be defined either by changes in the viscosity term, the domain size or the forcing function term.

Data generation For the data generation of DNS, we use a 5 point stencil for the classical central difference scheme of the Laplacian operator. For the Jacobian, we use a second order accurate scheme proposed by Arakawa that preserves the energy, enstrophy and skew symmetry (Arakawa, 1966). Finally for solving the Poisson equation, we use a Fast Fourier Transform based solver. We discretize a periodic domain into 512 × 512 points for the DNS and uses a RK4 solver with ∆ t = 5e -3 . We obtain the ground truth LES by applying a spatial filtering operator on the DNS trajectories. We also down-sample temporal and spatial grid, thus obtaining LES trajectories with a timestep ∆ t = 5e -2 on a 64 × 64 grid.

We generate 8 environments with 16 trajectories per environments for training. Trajectories have been generated on a temporal horizon [0 , 1] for training and [0 , 2] for in-distribution evaluation. During adaptation, we adapt our model on 4 unseen environments in a one-shot manner, using only one trajectory per environment. We evaluate out-distribution performance on 32 new ICs on a time horizon [0 , 2] . We consider the following initial conditions:

Vorticity is linked to energy by the following equation :

B.5 Combined equation

We used the setting introduced in Brandstetter et al. (2023), but with the exception that we do not include a forcing term and add the 4 th order spatial derivative. The combined equation is thus described by the following PDE:

α, β, δ and γ are the parameters that are varied.

Data generation We used the spectral solver proposed in Brandstetter et al. (2022) to generate the solution. We sampled 1200 environments for training, by sampling uniformly within the ranges α ∈ [0 . 5 , 1] , β ∈ [0 , 0 . 5] , δ ∈ [0 , 1] and γ ∈ [0 , 1] . For each parameter instance, we sampled 16 trajectories, resulting in 19200 trajectories. We then evaluate it on 19200 new trajectories. The trajectories were generated with a spatial resolution of 256 on a temporal horizon [0 , 30] . We only keep 10 time-steps. For out-distribution evaluation, we sample 10 new environments from the same ranges of parameters, but for parameters not seen during training. We have access to one trajectory from each evaluation environment for adapting the model, and evaluate it on 16 new trajectories.

C Trajectory based formulation

In practice, F e is unavailable and we can only approximate it from discretized trajectories. As done in (Kirchmeyer et al., 2022), we use a trajectory-based formulation of Eq. (9). We consider a set of

trajectories discretized over a uniform temporal and spatial grid includes T ∆ t ( s ∆ s ) d s states, where d s is the spatial dimension. ∆ t and ∆ s represent respectively the temporal and spatial resolution. T and S are the temporal horizon and spatial grid size. Our loss writes as:

u e j ( s k , t l ) is the state value in the j th trajectory from environment e at the spatial coordinate s k and time t l ≜ l ∆ t . u e ( t ) ≜ [ u ( s 1 , t ) , . . . , u ( s ( S/ ∆ s ) ds , t )] T is the state vector in the j th trajectory from environment e over the spatial domain at time t and u e 0 is the corresponding IC.

D Additional results

We conduct a comprehensive ablation study across various datasets to assess the robustness of our method. Our investigation explores several key aspects, including the learnability of PDE parameters within our hybrid framework. We also examine the adaptability of context-based methods to new, unseen environments, with varying numbers of adaptation trajectories N ad . Then, we show the effectiveness of our model in terms of complexity and performance when varying the size of the context vector. Additionally, we study the impact of initialization strategies for adaptation parameters in our meta-learning framework. Finally, we demonstrate the ability of our method to adapt rapidly to unseen environments with minimal gradient updates.

D.1 PDE parameter estimation

Existing works combining hard physical constraints and data-driven components either learn the PDE parameters or assume availability of the real parameters for a single PDE instance (Yin et al., 2021). In our case, we are trying to learn the PDE parameters in an unsupervised manner for each environment. In section 4.3, we proposed two strategies for estimating the PDE parameters. We report in the Figure 7 the training and adaptation MAE of the PDE parameters pendulum equation, where the first strategy has been adopted. In Figure 10, we report the training and adaptation convergence for the gray-scott equation, where the second strategy has been applied.

3.5 × 101

MAE of Parameter Estimate

3.45 × 10-1

Mean Absolute Error (MAE)

3.4 × 10-1

3.35 × 10-1

3.3 × 10-1

3.25 × 10-1

3.2 × 10-1

0

000

5000

7500

10000

12500

15000

17500

20000

Epoch

Convergence ofPDE Parameter Estimation

4.35 × 10-1

MAE of Parametcr Estimate

4.3× 10-1

Mean Absolute Error (MAE)

4.25 × 10-1

4.2× 10-1

4.15 × 10-1

4.1× 10-1

4.05 × 10-1

4×10-1

0

200

400

600

800

1000

Epoch

Our framework successfully estimates PDE parameters during both training and adaptation steps for the Pendulum and Gray-Scott equations. Notably, the accuracy of our framework in predicting PDE parameters is heavily dependent on the initialization of physical parameters. The specific initialization values used for PDE parameters are detailed in Table E.1.

MAE of Parameter Estimate

Mean Absolute Error (MAE)

10~2

101

000

5000

7500

10000

12500

15000

17500

20000

Epoch

D.2 Number of adaptation trajectories

Context adaptation While our methods remain effective on the Gray-Scott and the Burgers equation when adapting to unseen environments, meta-learning frameworks perform poorly on the Kolmogorov flow equations. This is particularly due to (i) the complex dynamics and very different behaviors that appear, (ii) the diversity of behaviors observed within a same environment for different initial conditions. Thus, the number of trajectories for training and for adaptation need to be sufficiently large for learning such dynamics; one-shot learning is therefore a very complex task for such systems. We report in table 5 the results when increasing the number of adaptation trajectories for Kolmogorov flow when adapting only context vectors:

We remark that context based frameworks do not scale well when increasing the number of adaptation trajectories.

Low-rank adaptation To overcome this issue, we propose to take advantage of the low-rank adaptation formulation proposed, which enable a cost-efficient adaptation of our model with respect to adapting all the parameters of the model. Instead of adapting c e , we adapt parameters Ac e B . We report results in table 6:

D.3 Data-driven Parameter initializations

Initialization of adaptation parameters is essential for correctly adapting to environments. We study how initialization impacts the training of our model on the Gray-Scott and Burgers equation. In LoRa,

Convergence ofPDE Parameter Estimation

3× 10-3

MAE of Parameter Estimate

2× 10-3

Mean Absolute Error (MAE)

101

6×10-4

4× 10-4

200

400

600

800

1000

Epoch

it is typically advised to initialize A using a Gaussian distribution and set B = 0 . In table 7, we report the relative loss for different initialization, including LoRa initialization. On both Gray-Scott and Burgers equation, the orthogonal initialization performs best.

D.3.1 Code dimension

In CoDa, the authors advocate to choose relatively small code size, corresponding to the degree of variations defining each environment. In our case, we proposed datasets with more degree of variations than the datasets used by (Kirchmeyer et al., 2022), presenting much more differences, necessitating larger code dimension size. In table 8, we show the performance of our model on both Gray-Scott and Burgers equation when varying the code dimension for CoDA and our method and the increase in number parameters.

CoDa

GEPS

1200K

1296K

1000K

800K

686K

600K

400K

229K

200K

152K

80K

84K

91K

106K

135K

6

8

10

12

14

16

Dimension of the Context

D.3.2 Adaptation speed

One important property of meta-learning framework is their ability to adapt to unseen environments in few adaptation steps. Our framework allows fast adaptation to new environments, compared to a state-of the art method like CoDA. We report in Figure the convergence of the loss during adaption for both our method and CoDA. With our framework, we are able to adapt to new environments in

0.16

Ours

CoDA

0.14

0.12

MSE

0.10

Relative

0.08

0.06

0.04

0.02

0.00

0

200

400

600

800

1000

Epoch

less than 100 steps, compared to CoDA which needs 500 steps. For both runs, we used the same learning rate lr = 0 . 01 .

D.3.3 Out-range temporal horizon generalization

We evaluate the ability of the different adaptation mechanisms for out-range temporal horizon extrapolation. All models have been trained on a temporal horizon [0 , T ] . We thus evaluate the performance of our models when extrapolating outside the trained temporal horizon. For out-range extrapolation, models are evaluated on the horizon [ T, 2 T ] . We report in-distribution and outdistribution performance in table 9. Error accumulates over time and leads to lower performance outside the training horizon, but GEPS still outperforms existing baselines.

D.3.4 Training time and number of parameters

We report in table 10 the training time and number of parameters for each adaptive conditioning method for all datasets:

E Implementation details

The code has been written in Pytorch (Paszke et al., 2019). All experiments were conducted on a single GPU:NVIDIA RTX A5000 (25 Go). We estimate the computation time needed for development and the different experiments to approximately 200 days.

E.1 Architecture

Adaptation

A;diag(ce)Bi

—(4)n

=

+

2(t)

zi-1(t)

+

Op

Wi

-(4)an

L1(zβ(t),ce)

Li(zi-1(t),ce)

ODESOLVE

Gp

Ga

ue(t + 1)

E.2 Data-driven model

We implement the dynamical model G θ with the following architectures:

Pendulum: we used MLPs composed of 4 hidden layers of width 64.

Gray-Scott: 4-layer 2D ConvNet with 64-channel hidden layers, and 3 × 3 convolution kernels with circular padding.

Burgers: 4-layer 1D ConvNet with 64-channel hidden layers, and 7 × 7 convolution kernels with circular padding.

Kolmogorov flow: 4 Fourier layers with 12 modes and a width of 16.

For all networks, we use Swish activation layers. We use an RK4 solver for the Neural ODE.

E.3 Physical model

For each dataset, we assume availability of some prior knowledge of the studied dynamical system, given as hard constraints. We report in the table 11 the physical knowledge assumed, the number of training parameters and the values of the parameters used at initialization:

E.4 Optimizer

For all datasets, we use the Adam optimizer and ( β 1 , β 2 ) = (0 . 9 , 0 . 999) for both training and adaptation. For training, we used a learning rate scheduler which reduces the learning rate when the loss has stopped improving. We set the threshold to 0 . 01 with a patience of 250 epochs with respect to the training loss. The minimum learning rate is 1 e -5 .

E.5 Hyper-parameter details

We report the hyper-parameters used for each dataset in the table:

E.6 Baselines implementation

For all baselines, we followed the recommendations given by the authors.

ERMexperiments We use the following baselines for the ERM experiments:

MP-PDE : We implement MP-PDE as a 1-step solver, where the time-bundling and pushforward trick do not apply. We use 6 message-passing blocks and 64 hidden features. We build the graph with the 8 closest nodes. We use a learning rate of 1e -3 and a batch size of 16. We train for 5000 epochs on Burgers and Gray-Scott equations.

FNO : FNO is trained for 5 , 000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e-3. We used 12 modes and a width of 32 and 4 Fourier layers. We also use a step scheduler every 250 epochs with a decay of 0.5.

CNN : We implement a CNN as 1-step solver, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e -3 . We also use a step scheduler every 250 epochs with a decay of 0 . 5 .

GEPS : We implement a CNN as 1-step solver with a multi-environment setting, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e -3 . We also use a step scheduler every 250 epochs with a decay of 0 . 5 .

In-distribution and out-distribution experiments We use the following baselines for the generalization experiments:

LEADS : We implement one-per-env method of the multi-task learning framework LEADS. For all datasets, we train the model during 20000 epochs with a learning rate lr = 0 . 01 , with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. All other hyper-parameters are the same than the one used in the paper.

CAVIA : We adapt CAVIA's framework for learning dynamical systems. We train CAVIA during 20000 epochs with a inner learning rate lrinner = 0 . 1 and an outer learning rate lr outer = 0 . 001 . For all datasets, we train the model during 20000 epochs with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. We tested different inner step values, and fixed to 7 for all datasets. We followed the authors recommendations for the experiments.

FOCA : We implement a first-order gradient-based method called FOCA to learn dynamical systems. We train CAVIA during 20000 epochs with an inner learning rate lrinner = 1 . 0 and an outer learning rate lrouter = 0 . 001 . For all datasets, we train the model during 20000 with a step scheduler every 250 epochs. We used τ = 1 for the exponential moving average and 5 inner steps.

CODA : We implement CoDA, a meta-learning framework for learning dynamical systems. We implemented the l1 and l2 regularization of the CoDA framework. We train the model during 20000 epochs with a step scheduler every 250 epochs. For the size of the contexts, we first fixed the size following the authors recommendations, but we observed improved performance using the same dimension used with our framework.

APHYNITY : We implemented a multi-environment version of the APHYNITY framework. We learn physics for each environment and learn to correct the physics model using a datadriven model that corrects all environments. During adaptation, we tune all the parameters of the model. For all datasets, we train the model during 20000 epochs with a learning rate lr = 0 . 001 , with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets.

F Qualitative results

In this section, we show different visualization of the predictions made by our framework and compare with the different baselines.

F.1 In-distribution and out-distribution generalization for Burgers dataset

We provide in Figure 16 in-distribution and out-distribution generalization for the Burgers equation.

F.2 Out-distribution generalization for Gray-Scott dataset

We provide in Figure 17 out-distribution generalization for the Gray-scott equation.

F.3 In-distribution generalization for Kolmogorov dataset

We provide in Figure 17 in-distribution generalization for the Kolmogorov equation.

Model

In-Domain generalisation

Out-Domain generalisation

MMm

t=0.00

t=0.01

=0.02

=0.03

t=0.06

t=0.07

t=0.05

t0.09

t=0.08

Ground truth

100

150

200

200

250

GEPS

xxjn

100

150

250

200

250

CoDA

100

150

200

50

100

150

250

m

100

150

200

250

200

250

LEADS

100

051

200

200

0SZ

Model

Initial condition

Predicted trajectory

In-t

Out-t

to

=

=1←

Ground

truth

口

GEPS

口

!

CoDA

口

口

LEADS

Model

Initial condition

Predicted trajectory

In-t

Out-t

to

t=T-

>t=T'

Ground

truth

GEPS

CoDA

LEADS

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provided experimental results in section 3 putting in evidence the importance of adaptation mechanisms for learning parametric PDEs, while ERM tend to fail in a IVP. Despite working when considering a history, we show our method still outperforms such methods. We then propose a data-driven and physics-aware meta-learning framework for learning to generalize on unseen environments, showing strong performance compared to existing meta-learning works in 5.3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We adressed the limitations of our framework in the section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results are provided in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided details in Appendix E about the hyper-parameters used to train our model, including model architecture. We provided details for both experiments done in section 3 and 5.3. We also provided how baselines were trained for both experiments. We also included a pseudo-code for our meta-learning framework, in algorithm 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provided open access to our code and access to the code used to generate our data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimiser, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided in section B all details to generate the different environments for our datasets and the number of training and test trajectories generated, for both training and adaptation settings. We also provided in E all details concerning the choice of optimiser and learning schedulers. We provided in Table 4 the hyper-parameters used for each dataset.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We did provided error bars to show statistical significance of our experiments for the second set of experiments in section 5.3. We did not include it for section 3, this is particularly due to the large number of experiments done and the number of baselines used, which require extensive computational demand.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided in E the computer resources used for our experiments with the memory of the GPU used. We also provided time of execution associated for the main experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not use personal data and credits in the code will be properly mentioned when needed. We also provided all config files for reproducibility.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our work is applied to physical sciences with generated data, so the applications are not related to societal concerns such as disinformation or privacy and security considerations.

Guidelines: Our work is applied to physical sciences, so the applications are not related to societal concerns such as disinformation or privacy and security considerations.

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We used synthetic data. It is not intended for widespread use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [No]

Justification: We credited the corresponding authors for the packages, baselines and datasets used in the experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We released our code on GitHub.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines: We do not do crowdsourcing.

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not relevant here.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes to solve parametric PDEs with the introduction of context parameters. The low-rank design allows for rapid adaptation to unseen conditions, and the experiments show comparable or slightly better performance of the method than baselines.

STRENGTHS:
1. The utilization of context parameters facilitates efficient adaptation to novel environments while minimizing computational overhead.

WEAKNESSES:
1. The main architecture of the proposed model is similar to LoRA, despite the fact that the context parameters are learned jointly during the pre-training stage.  The core innovation lies more in the combination of existing techniques rather than introducing a fundamentally new approach. The motivation of the model design is not well established as well.
2. The experiment results seem not strong, especially for the Gray-Scott equation.
3. The paper briefly mentions the computational efficiency of the proposed method but lacks a detailed analysis. 
4. The sensitivity of the proposed method to various hyperparameters is not extensively discussed, such as the size of the context vector and the rank in the low-rank adaptation.

QUESTIONS:
1. Why Neural ODE is utilized to generate the full trajectory? Do the experiment settings require continuous-time inference?

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the PDE solver generalization and proposes the GEPS model based on a low-rank-based meta-learning strategy. Specifically, the authors config the PDE solver as a low-rank framework, where different environments correspond to domain-specific diagonal matrices. During adaption, only the diagonal matrices are trained. Further, they employ a hybrid framework for incorporating physics prior as a trainable module in the model beginning. The authors provide extensive in- and out-domain experiments and compare the model with diverse baselines.\n\nSTRENGTHS:\n-\tThis paper focuses on an important problem, which is PDE generalization.\n-\tThe proposed method is reasonable and performs well in most cases.\n-\tThis paper is well-written and clear.\n\nWEAKNESSES:\n1.\tThe technical contribution is quite limited.\n\nUtilizing shared backbone and domain-specific weights for multi-environment or multi-task learning is a widely used design. There are many adaptor-based models, such as [1]. As for the hybrid framework, I think the authors fail to elaborate on the motivation of the overall design. Why use the prior module at the beginning? What about using it at the end of the model or using it as an adaptor? Also, why does the physics module not use the same low-rank design as the data part?\n\n[1] Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021\n\nIn addition, fine-tuning the diagonal matrix is also related to SVD-PINN [2], which should also be included as a baseline.\n\n[2] Svd-pinns: Transfer learning of physics-informed neural networks via singular value decomposition, SSCI 2022\n\n2.\tAbout experiments\n\nSome simple but important baselines are missing, including the adaptor-based method and some generalizable backbones. Specifically, for the adaptor-based method, the authors can train different adaptors for different environments based on a shared backbone. As for the generalizable backbones, they can experiment with advanced neural operators, such as [1,2]. I mean that train a universal model with thes backbones (the PDE coefficients are concatenated with inputs) and directly generalize them to new PDEs or coefficients. These baselines should also be included in Section 3.2 experiments.\n\n[1] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024\n\n[2] Scalable Transformer for PDE surrogate modeling, NeurIPS 2023\n\nAs a supplement to the main results in Tables 2 and 3, the training time of different methods should be compared.\n\n3.\tComparison with ensemble in Figure 3.\n\nI think ensemble could be a strong baseline, where it performs closely to GEPS in Burgers. Maybe the authors should replace the backbone with more advanced Transformer-based methods.\n\n4.\tAbout the title “adaptive conditioning”.\n\nThis paper proposed a low-rank training strategy. Thus, I think the adaptive conditioning is unsuitable.\n\nQUESTIONS:\nAbout the ERM experiments, will the model know the correct PDE coefficient during training and adaption? Do these experimented ERM baselines use the same information as GEPS?\n\nLIMITATIONS:\nThey have discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, we propose a meta-learning method called GEPS, which utilizes an adaptation approach to generalize a PDE solver to unseen environments. This method demonstrates better generalization compared to classical ERM approaches.\n\nSTRENGTHS:\nThis model can adapt to a new environment \\( f^e \\) in one shot and predict the time trajectory of the corresponding PDE at inference time. Specifically, the proposed model outperforms other existing meta-learning methods both in in-domain and out-domain environments of the initial conditions, as well as in in-range and out-of-range (extrapolation) time horizons.\n\nWEAKNESSES:\nSection 3.2 provides the motivation for the proposed method, but I am curious about how this motivation relates to the methods proposed in Sections 3.3.2 and 3.3.3. It would be helpful to explain how the issues identified with existing methods in the motivation section led to the use of neural ODEs and the model structure presented in Figure 4. Additionally, considering the results in Tables 2 and 3, the improvement in error scale compared to existing meta-learning methods does not seem significant. Figures 14 and 15 also show minimal differences. Therefore, it would be beneficial to experimentally demonstrate other advantages or unique aspects of the proposed GEPS method beyond its accuracy.\n\nQUESTIONS:\n* The overall relative error scale of FNO and MPPDE in Figures 2 and 3 and Table 1 is around 1e-1 (even when the environment and trajectory are sufficient), which appears to be generally larger than the errors presented in the respective original papers. Is this due to the difficulty of the equations, or is there another reason for the relatively large errors?\n\n* I am not quite clear on how the physical knowledge insertion $g_p$ and the data augmentation $g_a$ in Equation (6) are each encoded. Could you provide a more detailed explanation of the encoding methods?\n\n* What is the significance of the order of $g_p$ and $g_a$? Would it cause problems if $g_a$ were applied before $g_p$?\n\nLIMITATIONS:\nSee above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose GEPS, a method for meta-learning neural solvers for parametric PDEs. Similarly to gradient-based meta-learning methods such as MAML, the authors formulate the problem as a two-step optimization problem: the goal of the first step is to learn a model that's able to easily adapt to new environments via learning context-dependent parameters in the second step. A data-driven and physics-based decomposition is also proposed. The authors evaluate on four canonical dynamical systems and demonstrate state-of-the-art performance.\n\nSTRENGTHS:\n- The paper is well-written and clear.\n- The proposed method is straightforward and effective. The low-rank parameterization applied to meta-learning for dynamical systems is novel, as far as I am aware.\n- Experiments are thorough and comparisons to other meta-learning models are detailed. Evaluated on four canonical dynamical systems, the results are compelling.\n\nWEAKNESSES:\n- More experiments investigating the physics-aware component of the model would be illluminating. It's mentioned in Section 3.3.3 for the method to incorporate incomplete physics information, e.g. incomplete set of terms or inexact coefficients. However, evaluations of the hybrid method are only done with full knowledge of the system coefficients.\n- More ablation studies could be useful to further clarify the importance of each component of the proposed model. For example, it's unclear to what extent the improved performance is due to the specific low-rank parameterization (Eqn 3).\n- It would also be helpful to include the number of parameters of all methods evaluated.\n\nQUESTIONS:\n- Why is it that for Burgers, the hybrid setting of GEPS performs worse than the purely data-driven setting? Does this suggest that the data-driven/physics-based decomposition (Eqn 6) is not expressive enough to handle certain PDEs?\n- In Section 3.3.3, two strategies for adapting PDE parameters to a new environment are proposed (lines 213-219): learning $c^e$ only or learning both $c^e$ and $\\theta^e_p$. Which strategy is used for Tables 2 and 3?\n\nLIMITATIONS:\nThe authors address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}