Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning

Authors: armand  kassaï koupaï
jorge  mifsut benet
yuan  yin
jean-noël  vittaut
patrick  gallinari
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:36:14

Abstract: Solving parametric partial differential equations (PDEs) presents significant challenges for data-driven methods due to the sensitivity of spatio-temporal dynamics to variations in PDE parameters. Machine learning approaches often struggle to capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety of trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs is essential and that among them, \textit{adaptive conditioning}, allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, a simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully data-driven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, forcing terms and solution domain. *Project page*: https://geps-project.github.io

Main paper: 
 introduction: 
 Solving parametric partial differential equations, i.e. PDE in which certain parameters-such as initial and boundary conditions, coefficients, or forcing terms-can vary, is a crucial task in scientific and engineering disciplines, as it plays a central role in enhancing our ability to model and control systems, quantify uncertainties, and predict future events (Cohen & Devore, 2015) . Neural networks (NNs) are increasingly employed as surrogate models for solving PDEs by approximating their solutions (Long et al., 2018; Lu et al., 2021; Li et al., 2021) . A primary challenge with these data-driven solvers is their ability to generalize across varying contexts of the physical phenomena. This is especially pronounced for dynamical systems, which can exhibit significantly different behaviors when subject to small changes in PDE parameters.The usual approach for solving a parametric PDE with neural networks involves sampling instances from the PDE parameter distribution (e.g., the PDE coefficients), and then sampling trajectories for each instance of the underlying PDE (Takamoto et al., 2022) . The training set thus consists of multiple trajectories per parameter instance, with the objective of generalizing to new instances. This approach aligns with the classical empirical risk minimization (ERM) framework: it assumes that the training dataset is large enough to represent the distribution of the dynamical system behaviors and that this distribution is i.i.d. (Bodnar et al., 2024) . However, given the complexity of dynamical systems and the variety of PDE terms and parameters, these assumptions are rarely met in practice.To address this difficulty, some methods relax these assumptions. For instance, certain approaches embed the PDE parameters as inputs, aiming to generalize to new dynamics (Brandstetter et al., 2023; Takamoto et al., 2023) . While this can alleviate the issue, it does not facilitate generalization beyond the training distribution. Other approaches focus on few-shot settings, where a pre-trained model is fine-tuned for each new context (Subramanian et al., 2023; Herde et al., 2024) . As we will see, fine-tuning NN solvers typically requires a substantial amount of data, when it is often scarce in practice. In general, given the complexity of physical phenomena, it is often unrealistic to expect a sample distribution that is sufficiently representative to enable reliable generalization to new instances of the same phenomenon.Our claim is that neural PDE solvers should be explicitly trained with an adaptive conditioning mechanism to enable generalization to new parameter instances. For practical applications, this mechanism should condition the network on new dynamics using only a small amount of data. We assume access to several environments governed by the same general PDE, each defined by a unique set of parameters, from which trajectories can be sampled. This assumption aligns with approaches in multi-task learning (Yin et al., 2022) and meta-learning (Wang et al., 2022; Kirchmeyer et al., 2022) , where network weights are conditioned on a context that encapsulates environment-specific information. Although various adaptive conditioning mechanisms have been proposed (Finn et al., 2017; Kirchmeyer et al., 2022; Wang et al., 2022) , they are often limited to specific neural network architectures (Karimi Mahabadi et al., 2021) and struggle to scale effectively with data size or with the number of environments. We then propose an adaptation method that can handle a diversity of physical dynamics, is compatible with various NN backbones, and scales effectively with data size and the number of environments.After introducing the problem in Section 2, we emphasize in Section 3 the need for adaptive conditioning to solve parametric PDEs and illustrate the limitations of traditional ERM based approaches. We then introduce GEPS, a low-rank 1 st -order gradient-efficient adaptation mechanism for learning parametric PDE solvers. At inference, GEPS adapts to a new unseen environment, by learning a compact context vector c e . This approach is instanciated in two representative settings: (i) purely agnostic approaches, where the solver is trained without any prior physical knowledge, and (ii) physics-aware approaches that combine differentiable numerical solvers with NN components in a hybrid framework. Finally section 5 compares GEPS with alternative adaptation baselines. Our contributions are as follows:• We demonstrate on example PDEs that adaptive conditioning generalizes to multiple and new contexts, while classical ERM approaches fail to handle parametric PDEs. • We propose an effective and scalable adaptive conditioning framework for learning neural solvers. It is based on a first-order meta-learning approach and leverages low-rank adaptation, enabling adaptation to unseen environments in a few shot context. This formulation is versatile enough to encompass both pure data-driven and physics-aware models. • We provide experimental evidence of the performance and versatility of the approach on representative families of parametric PDEs, incorporating changes in initial conditions, PDE coefficients, forcing terms, and domain definitions, thereby covering both in-distribution and out-of-distribution scenarios. 
problem description: 
 We consider parametric time-dependent PDEs and aim to train models that generalize across a wide range of the PDE parameters, including initial conditions, boundary conditions, coefficient parameters, and forcing terms. For a given PDE, an environment e is an instance of the PDE characterized by specific parameter values. We assume that all environments share common global features, such as the general form of the dynamics, while each environment e exhibits some unique and distinct behaviors. A solution u e (x, t) of the PDE in environment e satisfies the PDE: ∂u e (x, t) ∂t = F e u e , ∂u e ∂x , ∂ 2 u e ∂x 2 , . . . , µ, f , ∀x ∈ Ω, ∀t ∈ (0, T ]B(u e )(x, t) = 0, ∀x ∈ ∂Ω, ∀t ∈ (0, T ](2) u e (x, 0) = u e 0 , ∀x ∈ Ω (3)where F e is a function of the solution u e , its spatial derivatives, the PDE coefficients µ and forcing terms f (x, t); B is the boundary condition (e.g., spatial periodicity, Dirichlet, or Neumann) that must be satisfied at the domain boundary ∂Ω and u 0 is the initial condition (IC) sampled with a probability measure u 0 ∼ p 0 (.). Environment e is thus defined by a set of parameters ξ = {B, p 0 , µ, f }.The targeted task is dynamics forecasting, where a neural solver is auto-regressively rolled out on a temporal horizon t ∈ [0, T ]. The goal is to approximate the evolution operator F e with a neural solver G θ (u e (x, t)) parametrized by θ ∈ R d θ , capable of generalizing to various PDE instances ξ, both within (in-distribution) and outside (out-of-distribution) the training parameter distribution.Therefore, we posit the observation of a set of environments e, each characterized by its specific PDE parameters ξ; trajectories are sampled for each environment, each characterized by an IC u e 0 . We define E tr as the set of environments used to train our model. In each training environment, N tr trajectories are available D e tr = {u i (x, t)} Ntr i=1 . For the in-distribution evaluation, the model has already learned conditioning context parameters (described later in section 4.1) for each training environment and is simply tested on new trajectories from the same environments using the appropriate context. For out-of-distribution, the model is evaluated on trajectories from new environments from an evaluation set E ev and is adapted. We then assume access to N ad trajectories D e ev = {u i (x, t)} Nad i=1 from the new environments to adapt the network. In our experiments we consider a scarce data, few-shot scenario where N ad = 1. After adaptation, we test our model on new unseen trajectories from these evaluation environments E ev . This setting is illustrated in Figure 1 . Before introducing our method, we first aim to motivate the need for adaptive conditioning in learning to solve parametric PDEs, while illustrating the limitations of the classical ERM setting.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We adressed the limitations of our framework in the section 6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
motivations: erm versus adaptive approaches for parametric pdes: 
 The classical ERM approach learns a single model on the data distribution, assuming that the training dataset is large enough to approximate the true data distribution. In practice, data acquisition is often costly and even simple physical systems can demonstrate a large variety of behaviors due to changes in the parameters. This may lead to poor generalization especially with scarce data.To illustrate this, we will compare the performance and behavior of classical ERM approaches and of our adaptive conditioning method for solving parametric PDEs, on two example datasets: the 2D Gray-Scott and the 1D Burgers equations. In these experiments, we generate for each PDE a series of environments by sampling only the physical coefficients of the PDE ξ = {µ} (more details in Appendix B). We consider in-distribution generalization and out-of-distribution generalization (respectively in sections 3.1 and 3.2, both for an initial value problem (IVP), a common setting where the initial condition u 0 corresponds to the system state at one time t 0 only. We then consider an alternative setting, where the neural solver is conditioned over a sequence of past states instead of one state only (section 3.3), this is denoted as "temporal conditioning". For all experiments, reported results correspond to the averaged relative L2 loss on 32 unseen trajectories per environment.Let us first compare adaptive conditioning and ERM approaches, for in-distribution evaluation, when scaling the number of training environments and trajectories. The models are trained on a range of environments -corresponding to different coefficients of the underlying PDE -and evaluated on the same environments with different initial conditions. Here GEPS is implemented with a classical CNN, while the tests for ERM are performed with four different backbones: the same CNN as used for GEPS but without adaptive conditioning, FNO (Li et al., 2021) , MP-PDE (Brandstetter et al., 2023) and Transolver (Wu et al., 2024) . Additionally, we also compared with a reference foundation model "Poseidon" (Herde et al., 2024) . This model has been pre-trained on a variety of IVP PDE equations and is fine-tuned on our data. Poseidon being trained on 2D data is thus evaluated only for the Gray-Scott equation. For all the baselines, we consider the classical IVP setting where only one initial state is given as input.Scaling w.r.t. the number of environments. We first examine how the number of training environments affects generalization to unseen trajectories within the same range of environments. The models are trained on 4 and 1024 environments with 4 trajectories per environments corresponding to different initial conditions. The evaluation is performed on 32 new trajectories from the same set of environments. As shown in Figure 2 , Transolver, FNO, MP-PDE, and CNN fail to capture the diversity of behaviors and their performance stagnate when increasing the number of training environments. Non-conditioned methods are not able to capture the diversity of behaviors for several environments, regardless of the backbone used, when using only an initial state as input. Poseidon on its side behaves much better on Gray-Scott and is able to capture this diversity of dynamics.Our adaptive conditioning approach (GEPS on the figures) performs significantly better than all the baselines, outperforming also the large Poseidon foundation model. We can also observe that GEPS benefits from being trained on a large amounts of environments, as its generalization performance improves with the number of training environments. Figure 3 shows the same behavior as for the previous experiments: ERM approaches rapidly reach a plateau and do not capture the variety of behaviors, even with a large number of trajectories, while GEPS scales well and improves with the number of training samples per environment. We additionally make a comparison with a CNN model trained and evaluated separately for each environment. We plot the average of the models' scores (indicated as "average" on the figure). This is an upper-bound of the performance that could be obtained with ERM models trained from scratch (no pretraining as for Poseidon). Note that this requires as many models as environments and is not scalable. While this performs significantly better than training over all the environments, GEPS matches or surpasses this approach. The experiments highlight that non-conditioned ERM approaches are unable to learn multienvironments datasets for solving IVP, whereas adaptive conditioning approaches like GEPS exhibit strong generalization performance, scaling with the number of training trajectories and environments. So far we have considered the classical IVP setting with only one initial state provided at time t 0 . We consider now the situation where the model has access at t 0 to an history of past states and not to a single initial state only. This is a common setting for learning PDE solvers: this allows the model to infer information on the dynamics and represents a more favorable case for the ERM baselines. We report in table 1 the in-distribution (In-d) and out-distribution (Out-d) distribution performance of ERM methods and GEPS, for Burgers and Gray-Scott PDEs. We did not consider the Poseidon (Herde et al., 2024) model that has been pre-trained only for one initial state IVPs. For in-distribution results, all methods are trained on 4 environments and evaluated on 32 new trajectories from the same environments. For out-of-distribution, adaptation (GEPS) and fine-tuning (baselines) is done on 1 trajectory per environment; 4 new environments are sampled and evaluation is performed on 32 new trajectories. We consider three different history sizes: 3, 5 and 10. Considering past history helps improve all the models. GEPS and the baselines show close performance for in-distribution, while GEPS is an order of magnitude better than the baselines for out-of-distribution.  
geps method: 
 We introduce our framework for learning to adapt neural PDE solvers to unseen environments. It leverages a 1 st order adaptation rule and low-rank adaptation to a new PDE instance. We consider two settings commonly used for learning the PDE solvers. The first one leverages pure agnostic data-driven approaches, as already considered in section 3. The second one leverages incomplete physics priors and considers hybrid approaches that complement differentiable numerical solvers with deep learning components. The general framework is illustrated in Figure 5 . We aim to train a model G θ to forecast dynamical systems coming from multiple environments. We perform adaptation in the parameter-space: some parameters are shared across all the environments while others are environment-specific. Training consists in estimating the shared parameters and learning to condition the model on environment specific parameters. At test time, the shared parameters are frozen and adaptation is performed on the environment specific parameters only. This setting is common to adaptation based approaches (Zintgraf et al., 2019; Kirchmeyer et al., 2022) , however most of them do not scale to large problems while GEPS introduces an efficient scalable adaptation mechanism.Formulation We adapt the parameters of our model G θ using a low-rank formulation. Most deeplearning architectures can be decomposed into modules or layers -in our experiments we use MLPs, CNNs and FNOs. For simplification let us then consider a layer L i from G θ parameterized by a weight matrix W Li ∈ R din×dout . Adaptation to an environment is performed through a low-rank matrix ∆W e Li = A Li diag(c e )B Li , whereA Li ∈ R din×r , B Li ∈ R r×dout , c e ∈ R r. The weights of layer L i with the adaptation mechanism are then:EQUATIONwhere diag(c e ) is a diagonal matrix capturing environment specific information and A Li , B Li , W Li are shared parameter matrices across all environments. Ideally, we want c e to capture the number of degrees of variations for our environments. If for example our model G θ is an MLP, the adaptation mechanism for a linear layer L i corresponds to:z e i = (W Li + A Li diag(c e )B Li )z e i-1 + b 1 Li + b 2 Li c e (5)whereW Li , A Li , B Li , b 1 Li , b 2Li are the parameters of layer L i , shared across all environments. Only c e is specific to each environment, but shared across all the layers of the network (cf Fig. 5 ).Considering a low-rank adaption rule is popular in NLP, where large pre-trained models are adapted to new tasks by learning a low-rank matrix ∆W = AB added to the frozen weights W of the pre-trained model (Hu et al., 2022) . Our approach differs in two ways from this setting: (i) the model is learned from scratch without pretraining, i.e., we learn parameters {W , A, B, c e } jointly, (ii) during adaptation, we can adapt to new environments e ∈ E ev by optimizing only the context vector c e , where it is initialized as c e = ctr , with ctr the averaged value of contexts learned during training. We experimentally show in Appendix D.3 that the classical Gaussian parameter initialization proposed in LoRA is inefficient in our context.This meta-learning framework operates in two steps: Training and Adaptation at inference. The goal is to learn an initial starting point during the training stage using a sample of environments, allowing adaptation to a new environment by adjusting a small subset of parameters based on a limited data sample from the new environment. Unlike many meta-learning gradient based approaches, GEPS does not involve an inner loop and is a 1 st order method. During the adaptation phase, all the parameters except the context parameters c e are frozen and c e is learned from the new environment sample. This approach ensures rapid adaptation by keeping c e low-dimensional. For simplicity, we refer to parameters shared across environments as θ s and parameters specific to each environment as δθ e ≜ c e , and denote θ e = {θ s , δθ e }. The optimization problem can thus be formulated as follows: We separate training and adaptation steps into a training and an adaptation loop as described in the pseudo-code Algorithm 1.We consider here an alternative problem to the above agnostic formulation. We assume that part of the physics is modeled through a PDE equation and shall be complemented with a statistical module. This is a common situation in many domains where prior physical knowledge is available, but only in an incomplete form. We follow the formulation in Yin et al. ( 2021) were starting from a complete PDE, we assume that part of the equation is known and will be modeled with a differentiable solver, while it is complemented with a deep learning component for modeling the unknown part. We also assume that the coefficients of the known part of the PDE are unknown and shall be estimated. We thus aim at solving both a direct problem (the NN parameters) and an inverse problem (the PDE coefficients of the known PDE part). We consider dynamics for a given environment of the form:EQUATIONF e and R e respectively represent the known and unknown physics of environment e and H is a function combining the two components which is unknown in practice. As for G θ , our model of the evolution operator for this physics-aware setting, we will use a simple combination: EQUATIONL(θ, D e tr ) = N j=1 t∈I,x∈Ω ∥(G θ (u j (x, t)) -H(F e (x, t, u j (x, t)) , R e (x, t, u j (x, t)))∥ 2 2 dxdt (9)The NN component is adapted as in section 4.1 through a c e parameter context vector. For estimating the PDE coefficients, we considered two alternatives. One consists in using the learned code c e for adapting the physical parameters θ e p = θ p + W p c e , where θ p , W p are shared parameters across all environments. The second one directly learns the parameters θ e p for each environment by gradient descent on the loss function. In the first approach, only context vectors c e are learned while in the second approach, c e and θ e p are learned jointly during adaptation. The performance of the two methods are similar. For the experiments, we evaluated both and used the better-performing one. 
dynamical systems: 
 We performed experiments on four dynamical systems, including one ODE and 3 PDEs. The ODE models the motion of a pendulum, which can be subject to a driving or damping term. We consider a Large Eddy Simulation (LES) version of the Burgers equation, a common equation used in CFD where discontinuities corresponding to shock waves appear (Basdevant et al., 1986) . We additionally study two PDEs on a 2D spatial domain: Gray-Scott (Pearson, 1993) , a reaction-diffusion system with complex spatio-temporal patterns and a LES version of the Kolmogorov flow, a 2D turbulence equation for incompressible flows. For the pure data-driven approaches, we make no prior assumption on the underlying physics, while for the physics-aware hybrid modeling problem, we assume that the physical equation is partially known and that the deep learning component targets the modeling of the unknown terms (details on the known/unknown terms for each equation are provided in Appendix B). The setting is the classical IVP formulation when only one initial state u e 0 is provided.While in section 3 we compared GEPS with baselines ERM approaches and with a foundation model, our objective here is to assess the performance of GEPS w.r.t. alternative adaptation based approaches.We evaluate the model performance on two key aspects. .Implementation We used a standard MLP for the Pendulum equation, a ConvNet for GS and Burgers equations and FNO for the vorticity equation. All activation functions are Swish functions.We use an Adam optimizer over all datasets. Contrary to Section 3, we perform time-integration with a NeuralODE (Chen, 2018) with a RK4 solver, as it was done for other multi-environments frameworks for physical systems (Yin et al., 2022; Kirchmeyer et al., 2022) . Architectures and training details are provided in Appendix E.Baselines As baselines, for the pure data-driven problem, we consider four families of multienvironment approaches. The first one consists in gradient-based meta-learning methods with CAVIA (Zintgraf et al., 2019) and FOCA (Park et al., 2023) . The second one is a multi-task learning method for dynamical systems: LEADS (Yin et al., 2022) . The third one is a hyper-network-based metalearning method which is currently SOTA among the adaptation methods, CoDA (Kirchmeyer et al., 2022) . As for the hybrid physics-aware problem, we implemented a meta-learning formulation of the hybrid method APHYNITY (Yin et al., 2021) , where the adaptation is performed on the physical PDE coefficients only while the NN component is shared across all the environments. GEPS-Phy is our physics-aware model (section 4.3) were all the parameters, PDE coefficients and context vector c e are adapted at inference. We also implemented "Phys-Ad", where we use the same formulation as for the hybrid GEPS-Phy, but adaptation is performed on the coefficients of the physical component only while the neural network component is shared across all environments. All the baselines share the same network architectures than the ones used for GEPS, indicated in the implementation paragraph above. More details on baselines implementation are provided in Appendix E.6.We report results for in and out-of-distribution generalization in table 2 for both the data-driven and the hybrid settings. Across all datasets, our framework performs competitively or better than the baselines for the two settings. Our method is able to correctly adapt to new environments in an efficient manner, updating only context parameters c e . For the agnostic data-driven experiments, GEPS obtains the best results, although being on the same range of performance as other methods.The main differentiator of GEPS w.r. than the strong baseline CoDA. Concerning the hybrid learning problem, incorporating physical knowledge leads to better results on all datasets except Burgers, where the fully data-driven method performs better. GEPS-Phy is better on all the datasets but Gray-Scott for which APHYNITY baseline is best. For Kolmogorov, the baselines did not converged at training.To further illustrate the benefits of GEPS, we compare it to CoDA, the SOTA adaptation model, on a large dataset with a larger model than the ones used for the previous experiments. We use a dataset generated from a PDE with multiple differentiable terms that encompasses several generic PDEs, inspired from Brandstetter et al. (2022) . The PDE writes as (more details in Appendix B):EQUATIONWe report the results in table 3. All the methods are trained using a ResNet (He et al., 2016) architecture, using a context size c e of size 8. For In-d, we trained our model on 1200 environments with 16 training trajectories per environment and evaluated it on 16 new trajectories. For Out-d, we further adapt each model to 10 new environments given 1 context trajectory per environment and then evaluate it on 16 new trajectories per environment. GEPS outperforms CoDA in terms of performance and number of parameters. This demonstrates the importance of scalable adaptation for few shot learning: GEPS is able to scale on large datasets using deep models while remaining parameter efficient.Question • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion and limitations: 
 Limitations We have seen that adaptation is essential for learning to generalize neural PDE solvers, and that within this setting, integrating physical knowledge may help. Adaptation still requires sufficient training samples -both environments and trajectories. These findings are still to be confirmed for more complex dynamics and real world conditions for which the variety of behaviors should be much larger than for the simple dynamics we experimented with.We empirically demonstrated the importance of adaptation for generalizing to new environments and its superiority with respect to ERM strategies. We proposed a new 1 st order and low-rank scalable meta-learning framework for learning to generalize time-continuous neural PDE solvers in a few-shot setting. We also highlighted the benefits of directly embedding PDE solvers as hard constraints in data-driven models when faced with scarce environments.We review data-driven methods for learning parametric PDEs and existing works for incorporating physical priors in neural networks, in the context of dynamical systems.Traditional ML paradigm The majority of current approaches for learning parametric PDEs considers a traditional ERM approach by sampling from the PDE parameter distribution. In MP-PDE (Brandstetter et al., 2023) , PDE parameters are directly embedded in the graph, allowing generalization to PDE parameters sampled from the same data distribution as those used for training.Neural operators (Kovachki et al., 2023; Takamoto et al., 2023) do similarly but require large training sets and often perform poorly for out-of-distribution (OOD) data. Alternatives to ERM such as invariant risk minimization (IRM) have shown stronger generalization by learning domain invariants (Arjovsky et al., 2020; Krueger et al., 2021) . However, even for simple classification problems, IRM methods can fail to capture natural invariances and are extremely sensitive to sampling (Kamath et al., 2021) .Multi-environment paradigm Several works have explored multi-environment/meta-learning settings for dynamical systems. 2022) used meta-learning with PINNs through context vectors c learned via auto-decoding. One drawback of PINN is their inability to handle scenarios where some physics are unknown.Data-driven methods with soft constraints Physics losses as proposed by (Raissi et al., 2019) have been used as prior knowledge under the form of soft constraints for neural operators, with the objective to alleviate data scarcity issues (Wang et al., 2021; Li et al., 2023) .Data-driven methods with hard constraints Alternatively, physical priors can be incorporated directly as hard constraints, leveraging the training of neural networks with differentiable physics. It has been particularly successful for accelerating computational fluid dynamics (CFD) by addressing numerical errors inherent in PDE discretization and for augmenting incomplete physics (Kochkov et al., 2021; Yin et al., 2021; Takeishi & Kalousis, 2021; Tathawadekar et al., 2023) . Most of these approaches do not consider the parametric PDE setting.We present the equations and the data generation settings used for all dynamical systems considered in this work. In table 4, we report the different PDE parameters changed to generate training and adaptation environments.We propose to study the damped and driven pendulum equation. The ODE represents the motion of a pendulum which can be subject to a damping and a forcing term. Even simple dynamical systems such as the pendulum equation can present a variety of complicated behaviors (e.g., under-damped, critically damped, resonance, super/sub-harmonic resonance, etc.), presented in figure 6 . The form of the ODE is:ν ∈ {5e-1, 5e-2, 5e-4} ν ∈ {1.0, 5e-5} f (x, t) = {F (sin(w f x) + cos(w f t)), 0} f (x, t) = F exp(-w f x 2 ) w f = 1.5 w f = {1.5, 3.0} 2D Gray-Scott F ∈ {0.03, 0.039} F ∈ {0.025, 0.042} k ∈ {0.058, 0.062} k ∈ {0.050, 0.065} 2D Kolmogorov Flow ν ∈ {1e -3, 1e -4} ν ∈ {5e -4} f (x, y, t) = {F sin(w f y), 0} {F exp(w f y 2 ), F (cos(w f x) + sin(w f t)} Domain = {[0, 0.75π], [0, π]} Domain = {[0, 1.25π]}EQUATIONwhere θ(t) is the angle, ω 0 the natural frequency, α the damping coefficient, f (t) is a forcing function of the form f (t) = F cos(w f t), where w f is the forcing frequency and F the amplitude of the forcing. In our case where we suppose we only have incomplete knowledge of the phenomenon, we consider we do not know the damping term. For the initial condition, the pendulum is dropped from a distribution θ(t 0 ) ∼ U(0, π/12) and dθ dt (t 0 ) ∼ U(0, 1).Data generation Each environment is defined by changes in the system parameters resulting in different behaviors. In training environments, the pendulum is either subject to damping or forcing, e.g., environment 1 is a damped pendulum (F = 0) while environment 2 is a driven pendulum (α = 0). We generate trajectories using a Runge-Kutta 8 solver. For training, we generated 4 distinct environments, each composed of 8 trajectories on the time horizon [0, 25] with a time step ∆t = 0.5.We also generate 32 trajectories on a longer time horizon [0, 50] per training environment to evaluate in-distribution generalization. For adaptation, we evaluate our method on 4 distinct environments defined by parameters unseen during training. Only one trajectory is generated per environment with a time horizon is [0, 25] to adapt the model to new dynamics; we evaluate the model's performance on 32 trajectories per environment on a time horizon [0, 50]. During adaptation, environments are defined such that trajectories are subject both to the forcing and damping term simultaneously.The PDE describes reaction-diffusion system with complex spatiotemporal pattern through the following 2D PDE:EQUATIONwhere u, v represent the concentrations of two chemical components in the spatial domain S with periodic boundary conditions. D u , D v denote the diffusion coefficients respectively for u, v and F, k are the reactions parameters. We consider complete knowledge of the physical model, but instead uses a RK4 solver with ∆t = 1 instead of using an adaptive RK45 solver. Burgers' equation is a nonlinear equation which models fluid dynamics in 1D and features shock formation:EQUATIONwhere u is the velocity field and ν is the diffusion coefficient and f (x, t) is a forcing term. The unresolved scales R closure (ū, u) and the forcing function is typically unknown, and needs to be directly learned from data. For the experiments done in Section 3, environments have been generated by sampling uniformly ν ∈ [1e -4, 0.5]. For experiments in Section 5.3, we generated environments by changing the diffusion coefficient or changing the forcing function, as detailed in table 4 .Data generation For the DNS, we generate complex trajectories using a 5 th order central difference scheme using Runge-Kutta 45 solver with a time-step ∆t = 1e-5 and ∆x = 2π 16384 . Such trajectories are particularly costly to generate, therefore, we rather use LES. To obtain the ground truth LES trajectories, we apply a spatial filtering operator on the DNS trajectories. We also down-sample the temporal and spatial grid. Therefore, we obtain LES trajectories with a timestep ∆t = 1e-3 and ∆x = 1 256 . For training, we generate 6 environments with 4 trajectories per environment. Trajectories have been generated on a temporal horizon [0, 0.05] for training and for in-distribution evaluation. For adaptation, we adapt our model on 4 new unseen environments using only trajectory per environment. We evaluate out-distribution performance on 32 new ICs on a time horizon [0, 0.05]. For the generation of the data, we define a [0, 2π] periodic domain and consider the following initial condition:E(k) = 2 3 √ π k k 0 4 1 k 0 exp - k k 0 2 (15)Velocity is linked to energy by the following equation :EQUATIONWe propose a 2D turbulence equation. We focus on analyzing the dynamics of the vorticity variable. The vorticity, denoted by ω, is a vector field that characterizes the local rotation of fluid elements, defined as ω = ∇ × u. Like Burgers, we generate LES, leading to an unknown unclosed term appearing in the vorticity equation, expressed as:EQUATIONHere, u represents the fluid velocity field, ν is the kinematic viscosity with ν = 1/Re and f (x, y, t) is a forcing term. The unresolved scales R closure (ū, u) and the forcing function is typically unknown, and needs to be directly learned from data. For the vorticity equation, environments can be defined either by changes in the viscosity term, the domain size or the forcing function term.Data generation For the data generation of DNS, we use a 5 point stencil for the classical central difference scheme of the Laplacian operator. For the Jacobian, we use a second order accurate scheme proposed by Arakawa that preserves the energy, enstrophy and skew symmetry (Arakawa, 1966) .Finally for solving the Poisson equation, we use a Fast Fourier Transform based solver. We discretize a periodic domain into 512 × 512 points for the DNS and uses a RK4 solver with ∆t = 5e -3. We obtain the ground truth LES by applying a spatial filtering operator on the DNS trajectories. We also down-sample temporal and spatial grid, thus obtaining LES trajectories with a timestep ∆t = 5e -2 on a 64 × 64 grid.We generate 8 environments with 16 trajectories per environments for training. Trajectories have been generated on a temporal horizon [0, 1] for training and [0, 2] for in-distribution evaluation. During adaptation, we adapt our model on 4 unseen environments in a one-shot manner, using only one trajectory per environment. We evaluate out-distribution performance on 32 new ICs on a time horizon [0, 2]. We consider the following initial conditions:E(k) = 4 3 √ π k k 0 4 1 k 0 exp - k k 0 2 (18)Vorticity is linked to energy by the following equation :EQUATIONWe used the setting introduced in Brandstetter et al. ( 2023), but with the exception that we do not include a forcing term and add the 4 th order spatial derivative. The combined equation is thus described by the following PDE:EQUATIONEQUATIONα, β, δ and γ are the parameters that are varied.Data generation We used the spectral solver proposed in Brandstetter et al. (2022) to generate the solution. We sampled 1200 environments for training, by sampling uniformly within the ranges α ∈ [0.5, 1], β ∈ [0, 0.5], δ ∈ [0, 1] and γ ∈ [0, 1]. For each parameter instance, we sampled 16 trajectories, resulting in 19200 trajectories. We then evaluate it on 19200 new trajectories. The trajectories were generated with a spatial resolution of 256 on a temporal horizon [0, 30]. We only keep 10 time-steps. For out-distribution evaluation, we sample 10 new environments from the same ranges of parameters, but for parameters not seen during training. We have access to one trajectory from each evaluation environment for adapting the model, and evaluate it on 16 new trajectories.In practice, F e is unavailable and we can only approximate it from discretized trajectories. As done in (Kirchmeyer et al., 2022) , we use a trajectory-based formulation of Eq. ( 9). We consider a set of trajectories discretized over a uniform temporal and spatial grid includes T ∆t ( s ∆s ) ds states, where d s is the spatial dimension. ∆t and ∆s represent respectively the temporal and spatial resolution. T and S are the temporal horizon and spatial grid size. Our loss writes as:EQUATIONu e j (s k , t l ) is the state value in the j th trajectory from environment e at the spatial coordinate s k and time t l ≜ l∆t. u e (t) ≜ [u(s 1 , t), . . . , u(s (S/∆s) ds , t)] T is the state vector in the j th trajectory from environment e over the spatial domain at time t and u e 0 is the corresponding IC.We conduct a comprehensive ablation study across various datasets to assess the robustness of our method. Our investigation explores several key aspects, including the learnability of PDE parameters within our hybrid framework. We also examine the adaptability of context-based methods to new, unseen environments, with varying numbers of adaptation trajectories N ad . Then, we show the effectiveness of our model in terms of complexity and performance when varying the size of the context vector. Additionally, we study the impact of initialization strategies for adaptation parameters in our meta-learning framework. Finally, we demonstrate the ability of our method to adapt rapidly to unseen environments with minimal gradient updates.Existing works combining hard physical constraints and data-driven components either learn the PDE parameters or assume availability of the real parameters for a single PDE instance (Yin et al., 2021) . In our case, we are trying to learn the PDE parameters in an unsupervised manner for each environment. In section 4.3, we proposed two strategies for estimating the PDE parameters. We report in the Figure 7 the training and adaptation MAE of the PDE parameters pendulum equation, where the first strategy has been adopted. In Figure 10 , we report the training and adaptation convergence for the gray-scott equation, where the second strategy has been applied. Context adaptation While our methods remain effective on the Gray-Scott and the Burgers equation when adapting to unseen environments, meta-learning frameworks perform poorly on the Kolmogorov flow equations. This is particularly due to (i) the complex dynamics and very different behaviors that appear, (ii) the diversity of behaviors observed within a same environment for different initial conditions. Thus, the number of trajectories for training and for adaptation need to be sufficiently large for learning such dynamics; one-shot learning is therefore a very complex task for such systems. We report in table 5 the results when increasing the number of adaptation trajectories for Kolmogorov flow when adapting only context vectors: CoDA 6.23e-1 6.22e-1 6.22e-1 6.20e-1 CAVIA 6.42e-1 6.40e-1 6.33e-1 6.29e-1 GEPS 5.22e-1 5.22e-1 5.21e-1 5.20e-1We remark that context based frameworks do not scale well when increasing the number of adaptation trajectories.Low-rank adaptation To overcome this issue, we propose to take advantage of the low-rank adaptation formulation proposed, which enable a cost-efficient adaptation of our model with respect to adapting all the parameters of the model. Instead of adapting c e , we adapt parameters Ac e B. We report results in table 6: Initialization of adaptation parameters is essential for correctly adapting to environments. We study how initialization impacts the training of our model on the Gray-Scott and Burgers equation. In LoRa, it is typically advised to initialize A using a Gaussian distribution and set B = 0. In table 7, we report the relative loss for different initialization, including LoRa initialization. On both Gray-Scott and Burgers equation, the orthogonal initialization performs best. In CoDa, the authors advocate to choose relatively small code size, corresponding to the degree of variations defining each environment. In our case, we proposed datasets with more degree of variations than the datasets used by (Kirchmeyer et al., 2022) , presenting much more differences, necessitating larger code dimension size. In table 8, we show the performance of our model on both Gray-Scott and Burgers equation when varying the code dimension for CoDA and our method and the increase in number parameters.Figure 13 : Model size with respect to code dimension Table 8 : Relative MSE on the full trajectory with varying code dimension code size ↓ CoDA GEPS Burgers Gray-Scott Burgers Gray-Scott1 2.04e-2 9.72e-2 1.49e-2 1.10e-1 2 2.03e-2 8.73e-2 1.35e-2 7.12e-2 4 1.96e-2 9.29e-2 1.46e-2 5.65e-2 8 1.78e-2 9.38e-2 1.43e-2 7.27e-2 16 1.84e-2 1.01e-1 1.30e-2 5.82e-2One important property of meta-learning framework is their ability to adapt to unseen environments in few adaptation steps. Our framework allows fast adaptation to new environments, compared to a state-of the art method like CoDA. We report in Figure the convergence of the loss during adaption for both our method and CoDA. With our framework, we are able to adapt to new environments in Figure 14 : Convergence speed of the model G θ to adapt to new environments for the Burgers dataset less than 100 steps, compared to CoDA which needs 500 steps. For both runs, we used the same learning rate lr = 0.01.We evaluate the ability of the different adaptation mechanisms for out-range temporal horizon extrapolation. All models have been trained on a temporal horizon [0, T ]. We thus evaluate the performance of our models when extrapolating outside the trained temporal horizon. For out-range extrapolation, models are evaluated on the horizon [T, 2T ]. We report in-distribution and outdistribution performance in table 9 . Error accumulates over time and leads to lower performance outside the training horizon, but GEPS still outperforms existing baselines.Table 9 : Generalization results for in-distribution and out-distribution environments -Test results for out-range time horizon (Out-t). Metric is the relative L2 loss. '-' indicates inference has diverged.Type ↓ Dataset → Pendulum Gray-Scott Burgers Kolmo Model ↓ In-d Out-d In-d Out-d In-d Out-d In-d Out-d LEADS 6.30e-1 7.28e-1 1.45e-1 1.27e-1 4.55e-2 3.22e-1 9.80e-1 1.01Data-driven CAVIA 40.4 1.88 3.73e-1 5.51e-1 6.44e-2 6.72e-1 1.06 1.06 FOCA 14.0 1.08 3.56e-1 3.34e-1 3.12e-1 4.60e-1 1.0 1.05 CoDA 9.54e-1 9.75e-1 1.52e-1 1.19e-1 3.41e-2 3.35e-1 8.29e-1 9.99e-1 GEPS 4.92e-1 8.45e-1 8.37e-2 6.91e-2 2.23e-2 2.68e-1 7.70e-1 9.74e-1 Hybrid APHYNITY 99.5 1.32 6.05e-3 4.19e-3 1.12e-1 8.09e-1 --GEPS-Phy 2.88e-1 7.12e-1 2.38e-2 1.92e-2 4.72e-2 3.59e-1 7.33e-1 9.32e-1We report in table 10 the training time and number of parameters for each adaptive conditioning method for all datasets: The code has been written in Pytorch (Paszke et al., 2019) . All experiments were conducted on a single GPU:NVIDIA RTX A5000 (25 Go). We estimate the computation time needed for development and the different experiments to approximately 200 days. We implement the dynamical model G θ with the following architectures:• Pendulum: we used MLPs composed of 4 hidden layers of width 64.• Gray-Scott: 4-layer 2D ConvNet with 64-channel hidden layers, and 3 × 3 convolution kernels with circular padding.• Burgers: 4-layer 1D ConvNet with 64-channel hidden layers, and 7 × 7 convolution kernels with circular padding.• Kolmogorov flow: 4 Fourier layers with 12 modes and a width of 16.For all networks, we use Swish activation layers. We use an RK4 solver for the Neural ODE.• CNN: We implement a CNN as 1-step solver, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e -3. We also use a step scheduler every 250 epochs with a decay of 0.5. • GEPS: We implement a CNN as 1-step solver with a multi-environment setting, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e -3. We also use a step scheduler every 250 epochs with a decay of 0.5.In-distribution and out-distribution experiments We use the following baselines for the generalization experiments:• LEADS: We implement one-per-env method of the multi-task learning framework LEADS. For all datasets, we train the model during 20000 epochs with a learning rate lr = 0.01, with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. All other hyper-parameters are the same than the one used in the paper. • CAVIA: We adapt CAVIA's framework for learning dynamical systems. We train CAVIA during 20000 epochs with a inner learning rate lr inner = 0.1 and an outer learning rate lr outer = 0.001. For all datasets, we train the model during 20000 epochs with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. We tested different inner step values, and fixed to 7 for all datasets. We followed the authors recommendations for the experiments. • FOCA: We implement a first-order gradient-based method called FOCA to learn dynamical systems. We train CAVIA during 20000 epochs with an inner learning rate lr inner = 1.0 and an outer learning rate lr outer = 0.001. For all datasets, we train the model during 20000 with a step scheduler every 250 epochs. We used τ = 1 for the exponential moving average and 5 inner steps. • CODA: We implement CoDA, a meta-learning framework for learning dynamical systems.We implemented the l1 and l2 regularization of the CoDA framework. We train the model during 20000 epochs with a step scheduler every 250 epochs. For the size of the contexts, we first fixed the size following the authors recommendations, but we observed improved performance using the same dimension used with our framework. • APHYNITY: We implemented a multi-environment version of the APHYNITY framework.We learn physics for each environment and learn to correct the physics model using a datadriven model that corrects all environments. During adaptation, we tune all the parameters of the model. For all datasets, we train the model during 20000 epochs with a learning rate lr = 0.001, with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets.In this section, we show different visualization of the predictions made by our framework and compare with the different baselines.We provide in Figure 16 in-distribution and out-distribution generalization for the Burgers equation.We provide in Figure 17 out-distribution generalization for the Gray-scott equation.We provide in Figure 17 in-distribution generalization for the Kolmogorov equation. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We provided experimental results in section 3 putting in evidence the importance of adaptation mechanisms for learning parametric PDEs, while ERM tend to fail in a IVP. Despite working when considering a history, we show our method still outperforms such methods. We then propose a data-driven and physics-aware meta-learning framework for learning to generalize on unseen environments, showing strong performance compared to existing meta-learning works in 5.3. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We did provided error bars to show statistical significance of our experiments for the second set of experiments in section 5.3. We did not include it for section 3, this is particularly due to the large number of experiments done and the number of baselines used, which require extensive computational demand.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provided in E the computer resources used for our experiments with the memory of the GPU used. We also provided time of execution associated for the main experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We do not use personal data and credits in the code will be properly mentioned when needed. We also provided all config files for reproducibility.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Our work is applied to physical sciences with generated data, so the applications are not related to societal concerns such as disinformation or privacy and security considerations.Guidelines: Our work is applied to physical sciences, so the applications are not related to societal concerns such as disinformation or privacy and security considerations.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We used synthetic data. It is not intended for widespread use.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [No]  
 
 Appendix:  


Decision: The paper proposes a novel method, GEPS, to improve the generalization of neural PDE solvers to unseen dynamics. GEPS employs a low-rank meta-learning strategy to enable efficient adaptation to new environments by adjusting only a small set of context parameters. The method incorporates physics knowledge through a hybrid framework, combining a physics module with a data-driven module. Extensive experiments demonstrate the effectiveness of GEPS in various spatio-temporal forecasting tasks.

Strengths:

The paper is easy to follow, with clear explanations and concise language.
The low-rank meta-learning strategy for adaptive conditioning is a novel contribution to the field of neural PDE solvers.
The authors provide extensive experiments to validate the effectiveness of GEPS, demonstrating its superiority over existing methods in many cases. 
The incorporation of physics knowledge through a hybrid framework offers a flexible approach to modeling complex PDEs.

Weaknesses:

While the low-rank meta-learning strategy is novel, the overall approach of using a hybrid framework and domain-specific weights is not entirely new.
The paper could provide a more detailed explanation of the motivation behind the specific design choices, such as the use of a prior module.
Some important baselines, such as adaptor-based methods and advanced neural operators, are missing from the experiments.

Recommendation:

The paper presents a promising approach for improving the generalization of neural PDE solvers. While the technical novelty is low, the paper's strengths in terms of clarity, empirical results, and hybrid framework outweigh the weaknesses. All the reviewers vote for acceptance. I, therefore, recommend accepting the paper and encourage the authors to use the feedback provided to improve the paper.

Reviews: review: paper_summary: paper_summary: The authors propose GEPS, a method for meta-learning neural solvers for parametric PDEs. Similarly to gradient-based meta-learning methods such as MAML, the authors formulate the problem as a two-step optimization problem: the goal of the first step is to learn a model that's able to easily adapt to new environments via learning context-dependent parameters in the second step. A data-driven and physics-based decomposition is also proposed. The authors evaluate on four canonical dynamical systems and demonstrate state-of-the-art performance.
strength_weakness: strength_weakness: - More experiments investigating the physics-aware component of the model would be illluminating. It's mentioned in Section 3.3.3 for the method to incorporate incomplete physics information, e.g. incomplete set of terms or inexact coefficients. However, evaluations of the hybrid method are only done with full knowledge of the system coefficients.
- More ablation studies could be useful to further clarify the importance of each component of the proposed model. For example, it's unclear to what extent the improved performance is due to the specific low-rank parameterization (Eqn 3).
- It would also be helpful to include the number of parameters of all methods evaluated.
questions: questions: - Why is it that for Burgers, the hybrid setting of GEPS performs worse than the purely data-driven setting? Does this suggest that the data-driven/physics-based decomposition (Eqn 6) is not expressive enough to handle certain PDEs?
- In Section 3.3.3, two strategies for adapting PDE parameters to a new environment are proposed (lines 213-219): learning $c^e$ only or learning both $c^e$ and $\theta^e_p$. Which strategy is used for Tables 2 and 3?
limitations: limitations: The authors address the limitations.
paper_summary: paper_summary: In this paper, we propose a meta-learning method called GEPS, which utilizes an adaptation approach to generalize a PDE solver to unseen environments. This method demonstrates better generalization compared to classical ERM approaches.
strength_weakness: strength_weakness: Section 3.2 provides the motivation for the proposed method, but I am curious about how this motivation relates to the methods proposed in Sections 3.3.2 and 3.3.3. It would be helpful to explain how the issues identified with existing methods in the motivation section led to the use of neural ODEs and the model structure presented in Figure 4. Additionally, considering the results in Tables 2 and 3, the improvement in error scale compared to existing meta-learning methods does not seem significant. Figures 14 and 15 also show minimal differences. Therefore, it would be beneficial to experimentally demonstrate other advantages or unique aspects of the proposed GEPS method beyond its accuracy.
questions: questions: * The overall relative error scale of FNO and MPPDE in Figures 2 and 3 and Table 1 is around 1e-1 (even when the environment and trajectory are sufficient), which appears to be generally larger than the errors presented in the respective original papers. Is this due to the difficulty of the equations, or is there another reason for the relatively large errors?

* I am not quite clear on how the physical knowledge insertion $g_p$ and the data augmentation $g_a$ in Equation (6) are each encoded. Could you provide a more detailed explanation of the encoding methods?

* What is the significance of the order of $g_p$ and $g_a$? Would it cause problems if $g_a$ were applied before $g_p$?
limitations: limitations: See above comments.
paper_summary: paper_summary: This paper focuses on the PDE solver generalization and proposes the GEPS model based on a low-rank-based meta-learning strategy. Specifically, the authors config the PDE solver as a low-rank framework, where different environments correspond to domain-specific diagonal matrices. During adaption, only the diagonal matrices are trained. Further, they employ a hybrid framework for incorporating physics prior as a trainable module in the model beginning. The authors provide extensive in- and out-domain experiments and compare the model with diverse baselines.
strength_weakness: strength_weakness: 1.	The technical contribution is quite limited.

Utilizing shared backbone and domain-specific weights for multi-environment or multi-task learning is a widely used design. There are many adaptor-based models, such as [1]. As for the hybrid framework, I think the authors fail to elaborate on the motivation of the overall design. Why use the prior module at the beginning? What about using it at the end of the model or using it as an adaptor? Also, why does the physics module not use the same low-rank design as the data part?

[1] Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021

In addition, fine-tuning the diagonal matrix is also related to SVD-PINN [2], which should also be included as a baseline.

[2] Svd-pinns: Transfer learning of physics-informed neural networks via singular value decomposition, SSCI 2022

2.	About experiments

Some simple but important baselines are missing, including the adaptor-based method and some generalizable backbones. Specifically, for the adaptor-based method, the authors can train different adaptors for different environments based on a shared backbone. As for the generalizable backbones, they can experiment with advanced neural operators, such as [1,2]. I mean that train a universal model with thes backbones (the PDE coefficients are concatenated with inputs) and directly generalize them to new PDEs or coefficients. These baselines should also be included in Section 3.2 experiments.

[1] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024

[2] Scalable Transformer for PDE surrogate modeling, NeurIPS 2023

As a supplement to the main results in Tables 2 and 3, the training time of different methods should be compared.

3.	Comparison with ensemble in Figure 3.

I think ensemble could be a strong baseline, where it performs closely to GEPS in Burgers. Maybe the authors should replace the backbone with more advanced Transformer-based methods.

4.	About the title “adaptive conditioning”.

This paper proposed a low-rank training strategy. Thus, I think the adaptive conditioning is unsuitable.
questions: questions: About the ERM experiments, will the model know the correct PDE coefficient during training and adaption? Do these experimented ERM baselines use the same information as GEPS?
limitations: limitations: They have discussed limitations.
paper_summary: paper_summary: This paper proposes to solve parametric PDEs with the introduction of context parameters. The low-rank design allows for rapid adaptation to unseen conditions, and the experiments show comparable or slightly better performance of the method than baselines.
strength_weakness: strength_weakness: 1. The main architecture of the proposed model is similar to LoRA, despite the fact that the context parameters are learned jointly during the pre-training stage.  The core innovation lies more in the combination of existing techniques rather than introducing a fundamentally new approach. The motivation of the model design is not well established as well.
2. The experiment results seem not strong, especially for the Gray-Scott equation.
3. The paper briefly mentions the computational efficiency of the proposed method but lacks a detailed analysis. 
4. The sensitivity of the proposed method to various hyperparameters is not extensively discussed, such as the size of the context vector and the rank in the low-rank adaptation.
questions: questions: 1. Why Neural ODE is utilized to generate the full trajectory? Do the experiment settings require continuous-time inference?
limitations: limitations: See weaknesses.

Comments: comment: We would like to sincerely thank you for your appreciation of our new experiments.

We understand your concern regarding the originality of the technical contributions. We hope that our main message regarding the use of a new inductive principle and the effective solution proposed will still prove valuable to the community in advancing parametric PDE neural solvers.

While our method consistently outperforms all baselines across the datasets, we acknowledge that the improvement in terms of performance is modest. Nonetheless, we believe it is noteworthy that our approach requires 20 times fewer adaptation parameters compared to the best-performing baseline, CoDA. Although this reduction in parameters is manageable for current models, we hope our work encourages the adoption of adaptation mechanisms, especially for foundation models where the computational cost is much greater and efficiency becomes crucial.
 
Thank you once again for the insightful discussion and for dedicating your time to help us strengthen our paper.
Many thanks to the authors for providing additional experiments and explanations. Although I still hold that the technical contribution remains modest, as the original low-rank adaptation method appears to perform comparably to the proposed approach in terms of both efficiency and effectiveness, I will raise my score to 5, as this paper reasonably introduces meta-learning for multi-environment dynamic system modeling.
We appreciate that our new experiments have addressed your concerns. We would also like to clarify the point you raised:

**Retraining the model to fit a new setting is not adaptive.**

We understand that the term "adaptive conditioning" might be confusing. In the meta-learning community, "adaptation" refers to the process where a model, trained on a range of environments, is efficiently retrained when exposed to data from a new distribution (or environment) to adjust to that new context.

This is the concept we aimed to convey in our title: rather than performing direct inference on data from new environments (which underperforms compared to our approach), parametric neural PDE solvers should incorporate an adaptation mechanism. This mechanism enables the model to quickly learn from minimal data in these new environments, thereby enhancing its generalization capabilities.

**This concept is too broad to accurately describe your method.**

We recognize the concern about the breadth of "adaptation." Our paper primarily highlights a limitation in existing methods for solving parametric PDEs and demonstrates that an adaptive mechanism can address this issue, independent of the specific method proposed. While we provide a concrete solution, our main contribution is more general.

However, if you believe the title should be more specific to our method, we can revise it to: Boosting Generalization in Parametric PDE Neural Solvers through Context-Aware Adaptation.

Thank you for the valuable feedback and the constructive discussion, which have helped us strengthen the quality of our paper.
I would like to thank your effort in providing new experiment results. It resolves my concern about baselines. 

I still hold concerns about "adaptive conditioning". In your definition, this concept is too wide to accurately describe your method. Also, retraining the model to fit a new setting is not "adaptive". I think the presentation of this paper needs a major revision. 

However, I appreciate your effort in providing new experiments and adding new baselines. I will raise my score to 5.
Dear reviewer,

Thank you again for the time taken to review our paper.

We would like to kindly remind that we have addressed your last concerns in our last response and hope you can take note of it this before the discussion deadline.

If so, we respectfully ask if you could kindly indicate whether your rating has been updated accordingly.

Thank you very much for your time, and we look forward to your response.
Dear reviewer,

As the discussion period comes to an end, we would like to express our gratitude once again for your time and for highlighting important concerns that we hope have been addressed.

You mentioned the absence of a baseline involving "a sharing backbone with domain-specific weights." We would like to kindly remind that we have addressed this concern in our last response and hope you can take note of it this before the discussion deadline.

If so, we respectfully ask if you could kindly indicate whether your rating has been updated accordingly.

Thank you very much for your time, and we look forward to your response.
We appreciate your positive feedback. Please don't hesitate to reach out if you have further suggestions or questions.
Thank you for taking time to read our answers to your concerns and acknowledging our added results. We would be happy to engage in a further discussion if you allow it.

The motivations served to point out a pitfall in current PDE solvers inductive principle formulation for tackling parametric PDEs. We thus proposed a new inductive principle formulation that could tackle this pointed challenge. As for the specific framework implementation, we aimed to introduce an effective solution applicable to a wide range of neural PDE solvers, supported by two key technical contributions:
- An effective yet computationally efficient meta-learning formulation, as demonstrated in Table 1 of the rebuttal, which compares favorably with existing frameworks.
- A new, flexible approach for learning hybrid models within a meta-learning framework, which, to the best of our knowledge, has not been shown before.

Being backbone agnostic and accepting physics if available allow it to handle a large number of PDE solvers encoutered in the literature.

Regarding the simple baseline « shared backbone and domain-specific weights », as mentioned to reviewer m65H, all our baselines are in fact "shared backbone with domain specific-weights", as shown in Figure 6 of [1] in appendix. We think multi-task and meta-learning frameworks, while different in their problem formulation, solve the same optimization problem, as shown in [2].

As per your requests, we added two simple strategies, where we have a shared backbone but have domain specific-weights for the first-layer or the last layer.

We report the results for Gray-Scott and Burgers for out-domain results (similar conclusions have been observed in-domain):

| Model | GS | Burgers |
|-------|---------|---------|
| first-layer     |    4.52e-2   |    7.34e-2   |
| last-layer     |    5.03e-2   |    8.03e-2   | 
| GEPS     |    1.86e-2   |    5.29e-2   | 

Concerning the results in Appendix C.2, we actually think this is one particular strength of our method, where we can also tune the low-rank matrices to obtain better accuracy at a reasonable cost if needed. This cost could be significant for large models or for larger number of adaptation samples, justifying why we first proposed to only tune a very small subset of parameters $c^e$.
 
We hope these results and the already provided experiments answer your concerns. Don’t hesitate to reach out if needed.

[1] Kirchmeyer et al., Generalizing to New Physical Systems via Context-Informed Dynamics Model. ICML 2022

[2] Wang et al., Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation. ICML 2021
Dear Reviewer, 

Please have a look at the last part of the author's rebuttal, the other reviews and indicate if your rating has been updated based on that.

Your Area Chair.
Thank you for your review and for appreciating our new experiments. We would like to engage in a further discussion if you allow it.

We agree that having a « shared backbone and domain specific weights » setting is an important baseline. However, as recent papers suggest [1], meta and multi-task learning, while conceptually different in the problem formulation, share the same optimization problem in reality.

All our baselines, whether meta-learning or multi-task learning, do have a shared backbone and domain specific weights, as you are suggesting. In CoDA [2], the paper provide a good analysis of the differences between all the different multi-environment learning frameworks using a shared backbone with domain specific weights, in Figure 6 in their supplementary material.

However, as per your request, we added two different baselines.

We proposed two different strategies:
- having a shared backbone but having domain specific weights for the first layer
- having a shared backbone but having domain specific weights for the last layer

We report the results for Gray-Scott and Burgers for out-domain results (similar conclusions have been observed in-domain):

| Model | GS | Burgers |
|-------|---------|---------|
| first-layer     |    4.52e-2   |    7.34e-2   |
| last-layer     |    5.03e-2   |    8.03e-2   | 
| GEPS     |    1.86e-2   |    5.29e-2   | 

We hope this and the already provided new experiments answer your concerns. We are happy to discuss more about it if needed.

[1] Wang et al., Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation. ICML 2021

[2] Kirchmeyer et al., Generalizing to New Physical Systems via Context-Informed Dynamics Model. ICML 2022
Thanks for the authors' responses. My concerns with regard to the efficiency are addressed.
However, while I agree with the statement that traditional “ERM” methods may struggle to capture the variability of complex physical systems, especially when only the initial conditions are provided (in which case it would be extremely challenging for the model to make accurate predictions, as different PDE coefficients or other components could lead to divergent future states), the relationship between the motivation and the specific model design is still unclear to me. The results in Appendix C.2 seem to indicate that the original low-rank adaptation method results in stronger performance without introducing much more computational overhead than the method proposed in this paper. Moreover, as suggested by other reviewers, I think that a simple baseline with "shared backbone and domain-specific weights" should be added. Therefore, I will maintain my current score.
Thanks for your response and new results. The experiments about Transformer-based models and efficiency helped me understand this method better. 

However, I still think the "shared backbone and domain-specific weights" setting is a strong and necessary baseline. Note that this baseline is not about meta-learning, it is just a classical multitask learning backbone. Besides, since the authors state that "adaptive conditioning" refers to "learning from multiple environments to adapt rapidly to a new one", the above "shared backbone and domain-specific weights" can also be viewed as a special "adaptive conditioning", where the domain-specific head can "adaptively" utilize the shared representations.

Thus, I decided to keep my original score.
Thanks to the authors for the detailed response and clarifications. I believe inclusion of these additional details will strengthen the paper. I will keep my score as is.
Thank you for your thoughtful review and for recognizing the additional experiments we included in response to your concerns.

We would like to emphasize again that, as indicated, our main contribution lies not solely in the technical advancements, but in highlighting the inadequacy of classical ERM for parametric PDEs and proposing an alternative inductive principle. We believe that this change in perspective represents a significant step forward that could impact recent trends in foundation models for PDEs, for example, those aimed at solving parametric PDEs.

In addition to this conceptual advancement, we introduced an effective solution applicable to a wide range of neural PDE solvers, supported by two key technical contributions:

- An effective yet computationally efficient meta-learning formulation, as demonstrated in Table 1 of the rebuttal, which compares favorably with existing frameworks.
- A new, flexible approach for learning hybrid models within a meta-learning framework, which, to the best of our knowledge, has not been shown before.

Respectfully, we would like to note that a score of 5 typically indicates a borderline acceptance, not a weak one.

We will incorporate your feedback into the revised version. Thank you again for your valuable insights, and we are happy to engage in further discussion.
Thank you for taking the time to explain my questions in detail and for showing the revised figures and tables. While I understand your explanation, I still find the motivation for the proposed method unclear (as pointed out by other reviewers regarding the lack of technical contributions). Additionally, in Tables 3 and 4, it seems there isn't a significant accuracy improvement compared to other benchmark methods, even with the error bars added from multiple experiments. Therefore, I will maintain my weak accept score of 5. However, I believe that the motivation for the proposed method and the advantages or unique aspects of the proposed GEPS method, which you explained to me, should be added and emphasized more in the revised version of the paper. Thank you.
The authors-reviewers discussion period has now started.

@Reviewers: Please read the authors' response, ask any further questions you may have or at least acknowledge that you have read the response. Consider updating your review and your score when appropriate. Please try to limit borderline cases (scores 4 or 5) to a minimum. Ponder whether the community would benefit from the paper being published, in which case you should lean towards accepting it. If you believe the paper is not ready in its current form or won't be ready after the minor revisions proposed by the authors, then lean towards rejection.

@Authors: Please keep your answers as clear and concise as possible.

The AC