Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning
Authors: Armand Kassaï Koupaï
Jorge Mifsut Benet
Yuan Yin
Jean-Noël Vittaut
Patrick Gallinari
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Learning
Parametric PDEs
Meta-Learning
physics-aware
Primary area: machine_learning_for_physical_sciences
TL;DR: We propose a $1^st$ order low-rank meta-learning model for generalizing Neural PDE dynamics solvers to unseen dynamics. Our approach is general, it can incorporate knowledge from known physics if available.

Abstract: Solving parametric partial differential equations (PDEs) presents significant challenges for data-driven methods due to the sensitivity of spatio-temporal dynamics to variations in PDE parameters. Machine learning approaches often struggle to capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety of trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs is essential and that among them, \textit{adaptive conditioning}, allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, a simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully data-driven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, forcing terms and solution domain. *Project page*: https://geps-project.github.io

GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning
Armand Kassaï Koupai - Jorge Mifsut Benet Jean-Noël Vittaut Patrick Gallinari!.4
Yuan Yin
Sorbonne Université, CNRS, ISTR, 75005 Paris, France 2 Valeo.ai, Paris, France
Sorbonne Université, CNRS, LIP6, 75005 Paris, France
Criteo Al Lab, Paris, France
Abstract
Solving parametric partial differential equations (PDEs) presents significant chal- lenges for data-driven methods due to the sensitivity of spatio-temporal dynamics variations in PDE parameters. Machine learning approaches often struggle capture this variability. To address this, data-driven approaches learn parametric PDEs by sampling a very large variety trajectories with varying PDE parameters. We first show that incorporating conditioning mechanisms for learning parametric PDEs essential and that among them, adaptive conditioning, allows stronger generalization. As existing adaptive conditioning methods do not scale well with respect to the number of parameters to adapt in the neural solver, we propose GEPS, simple adaptation mechanism to boost GEneralization in Pde Solvers via a first-order optimization and low-rank rapid adaptation of a small set of context parameters. We demonstrate the versatility of our approach for both fully data- driven and for physics-aware neural solvers. Validation performed on a whole range of spatio-temporal forecasting problems demonstrates excellent performance for generalizing to unseen conditions including initial conditions, PDE coefficients, foreing terms and solution domain. Project page: htass//gessprroeeetgituubu
Introduction
Solving parametric partial differential equations, i.e. PDE in which certain parameters- as initial and boundary conditions, coefficients, or forcing terms-can can vary, is a crucial task in scientific and engineering disciplines, it plays central role in enhancing our ability to model and control systems, quantify uncertainties, and predict future events (Cohen Devore 2015). Neural networks (NNs) are increasingly employed surrogate models for solving PDEs by approximating their solutions (Long et al.) 2018 Lu et al.) 2021; Li et al. 2021). A primary challenge with these data-driven solvers their ability to generalize across varying contexts of the physical phenomena. This is especially pronounced for dynamical systems, which can exhibit significantly different behaviors when subject small changes in PDE parameters.
The usual approach for solving a parametric PDE with neural networks involves sampling instances from the PDE parameter distribution (e.g., g,, the PDE coefficients), and then sampling trajectories for each instance of the underlying PDE (Takamoto et al. 2022). The training set thus consists of multiple trajectories per parameter instance, with the objective of generalizing new instances. This approach aligns with the classical empirical risk minimization (ERM) framework: it assumes that the training dataset is large enough to represent the distribution of the dynamical system behaviors and that this distribution i.i.d. (Bodnar et al. 2024). However, given the complexity of dynamical
Corresponding author: frmand.kassa @ isii..ppc..f
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

systems and the variety of PDE terms and parameters, these assumptions are rarely met in practice. To address this difficulty, some methods relax these assumptions. For instance, certain approaches embed the PDE parameters as inputs, aiming to generalize to new dynamics (Brandstetter et al. 2023 Takamoto et al., 2023). While this can alleviate the issue, it does not facilitate generalization beyond the training distribution. Other approaches focus on few-shot settings, where pre-trained model is fine-tuned for each new context (Subramanian et al., [023]] Herde et al. 2024). As we will see, fine-tuning NN solvers typically requires substantial amount of data, when it is often scarce in practice. In general, given the complexity of physical phenomena, it is often unrealistic to expect a sample distribution that is sufficiently representative to enable reliable generalization to instances of the same phenomenon.
Our claim is that neural PDE solvers should be explicitly trained with an adaptive conditioning mechanism to enable generalization to new parameter instances. For practical applications, this mechanism should condition the network on new dynamics using only small amount of data. We assume access to several environments governed by the same general PDE, each defined by a unique set of parameters, from which trajectories can be sampled. This assumption aligns with approaches in multi-task learning (Yin et al. 2022) and meta-learning (Wang et al. 2022, Kirchmeyer et al.) 2022). where network weights are conditioned on context that encapsu environment--peciiee information. Although various adaptive conditioning mechanisms have been proposed (Finn et al. 2017 Kirchmeyer et al., 2022) Wang al., (2022). they are often limited to specific neural network architectures (Karimi Mahabadi et al., (021) and struggle to scale effectively with data size or with the number of environments. We then propose an adaptation method that can handle a diversity of physical dynamics, is compatible with various NN backbones, and scales effectively with data size and the number of environments.
After introducing the problem in Section 2 we emphasize in Section 3 the need for adaptive conditioning to solve parametric PDEs and illustrate the limitations of traditional ERM based approaches. We then introduce GEPS, a low-rank 1*-order gradient-efficient adaptation mechanism for learning parametric PDE solvers. At inference, GEPS adapts to new unseen environment, by learning a compact context vector c". This approach is instanciated in two representative settings: (i) purely agnostic approaches, where the solver is trained without any prior physical knowledge, and (ii) physics-aware approaches that combine differentiable numerical solvers with NN components in hybrid framework. Finally section 5 compares GEPS with alternative adaptation baselines. Our contributions are as follows:
We demonstrate on example PDEs that adaptive conditioning generalizes to multiple and new contexts, while classical ERM approaches fail handle parametric PDEs.
We propose effective and scalable adaptive conditioning framework for learning neural solvers. It is based on first-order meta-learning approach and leverages low-rank adaptation, enabling adaptation to unseen environments in a few shot context. This formulation is versatile enough to encompass both pure data-driven and physics-aware models.
We provide experimental evidence of the performance and versatility of the approach on representative families of parametric PDEs, incorporating changes in initial conditions, PDE coefficients, forcing terms, and domain definitions, thereby covering both in-distribution and duits--dist scenarios.
2 Problem Description
We consider parametric time-dependent PDEs and aim to train models that generalize across wide range of the PDE parameters, including initial conditions, boundary conditions, coefficient parameters, and forcing terms. For given PDF, an environment is an instance of the PDE characterized by specific parameter values. We assume that all environments share common global features, such as the general form of the dynamics, while each environment exhibits some unique and distinct behaviors. A solution u (x, t) of the PDE in environment satisfies the PDE:
Ou Ou((x,)
u Ou Pu"
Fe
dx dz ......, ........f) Va E Ω Vt E (0,
(1)
Ot
Blu²) B(u²)(z, 1) = 0,
Vo E an, VL E (0,
(2)
u "x 0) = un.
Vx E Ω
(3)

---

where F is function of the solution u", its spatial derivatives, the PDE coefficients je and forcing terms f(x t): B is the boundary condition (e.g., spatial periodicity, Dirichlet, Neumann) that must be satisfied at the domain boundary 0 and ug is the initial condition (IC) sampled with probability measure tλg
pd (-). Environment thus defined by set of parameters
pB.p². p. f)
The targeted task is dynamics forecasting, where a neural solver is auto-regressively rolled out on [0, J. The goal is approximate the evolution operator ! with neural t)) parametrized by E Id,, capable of generalizing to various PDE instances both within (in- in-distribution) and outside cuutoo-disttibuttion the training parameter distribution.
temporal horizon
solver Go ut" (a, u=",;
Therefore, we posit the observation of set of environments each characterized by its specific PDE parameters E: trajectories are sampled for each environment, each characterized by an IC ug- We define &n the set of environments used to train model. In each training environment, Nir trajectories are available Dr {u.(I, ) For the in-distribution evaluation, the model has already learned conditioning context parameters (described later in section 4.1) for each training environment and is simply tested new trajectories from the same environments using the appropriate context. For out-of out-of-distrinee distribution the model is evaluated on trajectories from new environments from an evaluation set Eev and is adapted. from the environments adapt the network. In experiments few-shot scenario where Nat = 1. After adaptation, we test our model on new unseen trajectories from these evaluation environments Eev. This setting is illustrated in Figure T
then assume access to Nad trajectories DE, {u,( t) )
consider scarce data,
Training stage
Egp, with Nor trajectories per environnent
One-shot adaption
e; &
Trajectory
Time
Trijectoy
Naa
Figure 1: Multi-environment setup for the Kolmogorov PDE. The model is trained on multiple environments with several trajectories per environment (left). At inference, for new unseen environment it is adapted on one trajectory (right).
Before introducing our method, we first aim to motivate the need for adaptive conditioning in learning to solve parametric PDEs, while illustrating the limitations of the classical ERM setting.
3 Motivations: ERM versus adaptive approaches for parametric PDEs
The classical ERM approach learns single model on the data distribution, assuming that the training dataset is large enough to approximate the true data distribution. In practice, data acquisition is often costly and even simple physical systems can demonstrate a large variety of behaviors due to changes in the parameters. This may Iead to poor generalization especially with scarce data.
To illustrate this, we will compare the performance and behavior classical ERM approaches and of our adaptive conditioning method for solving parametric PDEs, on two example datasets: the 2D Gray-Scote Scott and the 1D Burgers equations. these experiments, we generate for each PDE a series of environments by sampling only the physical coefficients of the PDE & = { (more details in Appendix B We consider in in-distribution generalization and out-of-distribution generalization (respectively sections 3.1 and 3.2 both for an initial value problem (IVP), common setting where the initial condition uh corresponds to the system state one time lo only. We then consider an alternative setting, where the neural solver is conditioned over sequence of past states instead of

---

one state only (section 3.3). this is denoted as "temporal conditioning". For all experiments, reported results correspond to the averaged relative L2 loss on 32 unseen trajectories per environment.
3.1 In-distribution generalization for IVP: classical vs. adaptive conditioning approaches
Let us first compare adaptive conditioning and ERM approaches, for in-distribution evaluation, when scaling the number of training environments and trajectories. The models are trained on a range of environments - corresponding to different coefficients of the underlying PDE and evaluated on the same environments with different initial conditions. Here GEPS is implemented with classical CNN, while the tests for ERM are performed with four different backbones: the same CNN as used for GEPS but without adaptive conditioning, FNO (Li et al., 2021). MP-PDE (Brandstetter et al., 2023) and Transolver (Wu et al., 2024). Additionally, we also compared with a reference foundation model "Poseidon" (Herde et al. 2024). This model has been pre-trained
a variety of IVP PDE equations and is fine-tuned on our data. Poseidon being trained on 2D data is thus evaluated only for the Gray-Scott equation. For all the baselines, consider the classical IVP setting where only one initial state is given as input.
Scaling w.r.t the number of environments. We first examine how the number of training environments affects generalization to unseen trajectories within the same range of environments. The models trained on 4 and 1024 environments with 4 trajectories per environments corresponding to different initial conditions. The evaluation is performed on 32 new trajectories from the same set of environments. As shown in Figure 2] Transolver, FNO, MP-PDE, and CNN fail capture the diversity of behaviors and their performance stagnate when increasing the number of training environments. Non-conditioned methods are not able to capture the diversity of behaviors for several environments, regardless of the backbone used, when using only an initial state as input. Poseidon on its side behaves much better on Gray-Scott and is able capture this diversity of dynamics. Our adaptive conditioning approach (GEPS on the figures) performs significantly better than all the baselines, outperforming also the large Poseidon foundation model. We can also observe that GEPS benefits from being trained on large amounts of environments, as its generalization performance improves with the number of training environments.
Scont Equation
Borgees Equaticn
So-1
2012
162
340
4X1
100
1007
100
1000
MIVIE
180
-X-
Thunther
Powadne
Figure 2: Comparison of ERM approaches (shades of blue) and Poseidon foundation model (green) with our framework GEPS (red) when increasing the number of training environments.
Scaling w.r.t. the number of trajectories per training environment. For this second series of experiments, we fix the number of environments at 4 and vary the number of training trajectories per environment from 4 up to 1024. Figure shows the same behavior as for the previous experiments: ERM approaches rapidly reach a plateau and do not capture the variety of behaviors, even with a large number of trajectories, while GEPS scales well and improves with the number of training samples per environment. We additionally make comparison with a CNN model trained and evaluated separately for each environment We plot the average of the models' scores (indicated as "average" on the figure). This is an upper-bound of the performance that could obtained with ERM models trained from scratch (no pretraining as for Poseidon). Note that this requires as many models as environments and not scalable. While this performs significantly better than training over all the environments, GEPS matches or surpasses this approach.

---

Gray-Soon Equatin
Burgers Equaties
1-1
Tel
Se-2
162
CNN
MPPDE
END
Tunden
Averge
Poesker
+
CEPS
Figure 3: Comparison ERM approaches (shades blue) and Poseidon (green) with our framework GEPS (red) when increasing the number of trajectories per environment.
The experiments highlight that non-conditioned ERM approaches are unable to learn multi- environments datasets for solving IVP, whereas adaptive conditioning approaches like GEPS exhibit strong generalization performance, scaling with the number of training trajectories and environments.
3.2 Nut-of-distribution generalization to new environments for IVP: classical adaptive conditioning approaches
Let consider the cittof-ddistibuttoo behavior of the two approaches. The models trained on a sample of the environments from & and their associated trajectories, in the same condition as for section 3.1 They are then evaluated the trajectories of new environments. We report in figure the -ut-of-distribttion generalization performance of ERM methods and GEPS, for the Gray-Scott (left) and 1024 (right) environments, with trajectories
and Burgers equations, when pretrained per environment, For the test, one considers new environments and evaluate on 32 trajectories per environment. Adaptation (GEPS) or fine tuning (baselines) performed on one trajectory of new environment. As for the baselines, we consider CNN and Transolver, plus the Poseidon foundation model for the 2-D Gray-Scott equation only. As above, one may observe large performance gap between the non adaptive approaches and the adaptive GEPS. This supports our claim on the limitations of pure ERM based approaches to generalize to unseen dynamics configurations and new environments.
environments (Gray-Scott)
(Gray-Scott)
0.422
1
30
2x
0.296
4x-1--1
0.054
6x 10
G3PS
ROSOIDON
CNN
FSSDEER
3x.10
OCPS
POSCIDON
".SOLVER
CMN
Pretrained environments (Burgers)
Pretrained 1024
(Burgers)
0110
4x
3x.0
0278
6x
0.054
T:SIVVR
GPPS
7-SOKVER
Figure 4: Out-distribution generalization on 4 new environments using one trajectory per environment for fine-tuning or adaptation. Models have either been pretrained on environments (left column) or 1024 environments (right columns). Metric is Relative L2 loss.
3.3
In and out-of-distribution generalization performance with temporal conditioning
So far we have considered the classical IVP setting with only one initial state provided time 4g. We consider now the situation where the model has access at to to history of past states and not to
5

---

single initial state only. This is common setting for learning PDE solvers: this allows the model infer information on the dynamics and represents a more favorable case for the ERM baselines. We report in table | the in-distribution (In-d) and out-distribution (Out-d) distribution performance of ERM methods and GEPS, for Burgers and Gray-Scott PDEs. We did not consider the Poseidon (Herde et al., (024) model that has been pre-trained only for one initial state IVPs. For in-distribution results, all methods are trained on environments and evaluated on 32 new trajectories from the same environments. For out-of-distribution, adaptation (GEPS) and fine-tuning (baselines) is done trajectory per environment; 4 new environments are sampled and evaluation is performed on 32 new trajectories. We consider three different history sizes: 3, 5 and 10. Considering past history helps improve all the models. GEPS and the baselines show close performance for in n--distribution, while GEPS is order of magnitude better than the baselines for out-of- distribution.
Table In-distribution and out-distribution results comparing different history window sizes. Metric is the Relative L2 loss.
History
Method
In-d
Oui-d
In-d
Out-d
Ir-d
Ont-d
Bargers equation
Transolver
3.22e-1
1.12e-1
3.03e-1
5.64e.1
2.49e-1
FNO
4.28c-1
7.68c-1
3.07e-1
6.43e-1
6.13c-2
2,55c-1
CNN
1.8o-1
4.4e-1
1.62e-1
3.34e-1
3.16e-2
6.32e-2
GEPS
1.25e-2
1.63-22
8.61e.3
1,04e-3
6.14e.3
8.73e.3
Gray Scott equation
Transolver
1.82c-1
4.33c-1
9.88c-2
3.90e-1
9.57e-2
3.60e-1
1.86-1
4.67c-1
1.76e-1
3.87c-1
1.93e-
4.03c-1
CNN
7.12e-2
3.51e-1
5.96e-2
2.18e-1
6.54e-2
2.23e-1
GEPS
4.02e-2
5.78e-2
3.04e-2
5.02e-2
3.82e-2
5.14e-2
4 GEPS method
We introduce our framework for learning to adapt neural PDE solvers to unseen environments. leverages 1s order adaptation rule and low-rank adaptation two settings commonly used for learning the PDE solvers. The first one leverages pure agnostic data-driven approaches, already considered in section 3 The second one leverages incomplete physics priors and considers hybrid approaches that complement differentiable numerical solvers with deep learning components. The general framework illustrated in Figure 5
PDE instance. We consider
Adaptation
dlagge" B
W
Ly(u).c))
L, 2.c5)
4nt0h 2.5")
Figure 5: Our adaptation framework for our data-driven model. Block in blue refers to the data-driven module G.. Blocks L; in pink refer the trainable modules. The green block describes the adaptation mechanism for the data-driven component, with Wz, the weights of layer L;- Context vector e conditions all the layers L..
4.1 Adaptation rule
We aim to train a model Go forecast dynamical systems coming from multiple environments. We perform adaptation in the parameter-space: some parameters are shared across all the environments while others are environment--peccff.. Training consists in estimating the shared parameters and learning to condition the model environment specific parameters. At test time, the shared

---

parameters are frozen and adaptation is performed on the environment specific parameters only. This setting is common adaptation based approaches (Zintgraf et al.) 2019 Kirchmeyer al. (002)). however most of them do not scale to large problems while GEPS introduces an efficient scalable adaptation mechanism.
Formulation We adapt the parameters of our model Go using low-rank formulation. Most deep- learning architectures can be decomposed into modules or layers in our experiments we use MLPs, CNNs and FNOs. For simplification let us then consider a layer L; from Go parameterized by weight matrix WT, E Rd xdout, Adaptation to an environment is performed through a low-rank matrix AWE L; Ar, diag(c")BL. where AL, E Rdaxr, BL. E IRrxdous, of layer L, with the adaptation mechanism are then:
R. The weights
W = WL, + AL, BL,
where (iag (c*) is a diagonal matrix capturing environment specific information and AL., BL,, WL, shared parameter matrices across all environments. Ideally, want et to capture the number of degrees of variations for our environments, for example our model Go is an MLP, the adaptation mechanism for linear layer L; corresponds to:
(Wz, + AL, diag(( Bu.))(( Pt, bb c
(5)
where W 1. Ar., Br. b/. b2. the parameters of layer L,, shared across all environments. Only c is specific to each environment, but shared across all the layers of the network (cf Fig. 5
Considering low-rank adaption rule is popular in NLP, where large pre-trained models are adapted to new tasks by learning a low-rank matrix AW = AB added to the frozen weights W of the pre-trained model (Hu et al. 2022). Our approach differs in two ways from this setting: (i) the model learned from scratch without pretraining. i.e., we learn parameters {W, A, B, e²) jointly, (ii) during adaptation, we can adapt to new environments vector c", where it is initialized as c = Cu. with čr the averaged value of contexts learned during training. We experimentally show in Appendix(D.3) that the classical Gaussian parameter initialization proposed in LoRA is inefficient in our context.
&ev by optimizing only the context
4.2 Two-step training procedure
This meta-learning framework operates in two steps: Training and Adaptation inference. The goal to learn an initial starting point during the training stage using a sample environments, allowing adaptation to a new environment by adjusting small subset of parameters based on limited data sample from the new environment. Unlike many meta-learning gradient based approaches, GEPS does not involve an inner loop and is 1st order method. During the adaptation phase, all the parameters except the context parameters c are frozen and c is learned from the new environment sample. This approach ensures rapid adaptation by keeping e low-dimensional. For simplicity, refer to parameters shared across environments as 6" and parameters specific to each environment as 88 A c², and denote g = -0*,000 5.".. The optimization problem can thus be formulated as follows:
min .68
Σ C((0", (80), ).0).D))
E
(6)
subject to 80 = arg min
Σ
C(10"800" 8,0 D.))
59c
DREE
We separate training and adaptation steps into training and an adaptation loop described in the pseudo-code Algorithm
4.3 Hybrid formulation for learning dynamics
We consider here an alternative problem to the above agnostic formulation. We assume that part of the physics modeled through PDE equation and shall be complemented with a statistical module. This is common situation in many domains where prior physical knowledge is available, but only in an incomplete form. We follow the formulation in Yin al. (202T) were starting from a complete PDE, we assume that part of the equation is known and will be modeled with differentiable solver,
7

---

Algorithm 1: Training and adaptation for our method
Training:
Input:
(Pitees, with # = Nu;
Choose proper initialization for 0", Assign eº +- 0, lr {0t, cc'}
while no convergence do
Sample D from Ueeg D
0 - ~ V [((0" D.)) :
end
Adaptation:
{ Eee with #D = 1: Load pre-trained weights 0", Assign t On while no convergence do
Input:
Sample De, from Usee, D end
c² f- c² V C(10", (.))
while it is complemented with a deep learning component for modeling the unknown part. We also assume that the coefficients of the known part of the PDE are unknown and shall be estimated. We thus aim at solving both a direct problem (the NN parameters) and an inverse problem (the PDE coefficients of the known PDE part). We consider dynamics for given environment of the form:
Ou (z, = H(F (u(x,t),,. t), (u(x, Ôt
.), Ve E Vt E (0,
F and Rd respectively represent the known and unknown physics of environment and H is function combining the two components which is unknown in practice. As for Gg, our model of the evolution operator for this physics-awan setting, we will use simple combination:
Gg = Ga o 9p
(8)
where Gn encodes the physical knowledge and corresponds to the known part the PDE physical model and G. is the data-driven model term complementing Gy- With this model, we use auto- regressive formulation to generate the full trajectory, using Neural ODE (Chen et al. 2018) as time-stepper for predicting the state uetr ue+r uo Jr Go(u(r)drr,
illustrated in Figure 15 The model is trained directly from trajectories simulated with the full PDE (known + unknown PDE components) using the MSE loss DE (more details Appendix C):
N
((),
Σ #=1 1ET,EE
[[ t)) - H(F" (x, t, ()) H' (z, (r, (9)
The NN component is adapted as in section 4.1 through a c parameter context vector. For estimating the PDE coefficients, we considered two alternatives. One consists in using the learned code c for adapting the physical parameters 8G 0p Wpc", where 0p. Wp shared parameters across all environments. The second one directly learns the parameters 8% for each environment by gradient descent on the loss function. In the first approach, only context vectors e learned while in the second approach, cf and B are learned jointly during adaptation. The performance the two methods are similar. For the experiments, evaluated both and used the better-performing one.
Experiments
5.1 Dynamical systems
We performed experiments on four dynamical systems, including one ODE and 3 PDEs. The ODE models the motion of pendulum, which can be subject to driving damping term. We consider a Large Eddy Simulation (LES) version the Burgers equation, a common equation used in CFD where discontinuities corresponding to shock waves appear (Basdevant et al. 1986). We additionally study two PDEs on 2D spatial domain: Gray-Scott (Pearson) 1993), reaction-diffusion system with complex ppatio-tempora patterns and a LES version of the Kolmogorov flow, 2D turbulence

---

equation for incompressible flows. For the pure data-driven approaches, we make no prior assumption on the underlying physics, while for the physics-aware hybrid modeling problem, we assume that the physical equation is partially known and that the deep learning component targets the modeling of the unknown terms (details on the known/unknown terms for each equation are provided in Appendix B) The setting is the classical IVP formulation when only one initial state ug is provided.
5.2 Evaluation Setting
we compared GEPS with baselines ERM approaches and with a foundation model, our objective here is to assess the performance of GEPS w.r.t. alternative adaptation based approaches. We evaluate the model performance on two key aspects. In-distribution generalization: the model capability to predict trajectories defined by unseen ICs on all training environments e E Eu, referred as In-d. Out-of-distributien generalization: the model ability to adapt to new environment e E Eer by predicting trajectories defined by unseen ICs, referred as Out-d. Each environment e E is defined by changes in system parameters, forcing terms or domain definition. dp represents the degrees of variations used to define an environment for each PDE equation; dp = 4 for the pendulum equation, dz = 3 for the ID Burgers, dp = 2 and d, 3 for the Gray-Scott and the Kolmogorov flow equation respectively (more details in Table Nrr trajectories per training environment. For adaptation, we consider Nud = 1 trajectory per new environment in & to infer the context vector ef. Evaluation is performed on 32 new test trajectories ||a, 11:1..
While in section 3
4 in Appendix [B). For each dataset, we collect
per environment. We report in Table the relative MSE: N E==1
11::1 3
5.3 Generalization results
Implementation We used a standard MLP for the Pendulum equation, a ConvNet for GS and Burgers equations and FNO for the vorticity equation. All activation functions are Swish functions. We use an Adam optimizer over all datasets. Contrary to Section we perform time-integration with a NeuralODE (Chen 2018) with a RK4 solver, as it was done for other multi nuuti-ennvionments frameworks for physical systems (Yin et al. 2022 Kirchmeyer et al. 2022). Architectures and training details are provided in Appendix
Baselines As baselines, for the pure data-driven problem, we consider four families of multi- enviromment approaches. The first one consists gradient- meta-learning methods with CAVIA (Zintgraf et al. 2019) and FOCA (Park et al. 2023). The second one is multi-task learning method for dynamical systems: LEADS (Yin et al. 2022). The third one is a hyper-network- neter---oookk- meta- learning method which is currently SOTA among the adaptation methods, CoDA (Kirchmeyer et al., 2022). As for the hybrid physics-awan problem, we implemented meta-learning formulation of the hybrid method APHYNITY (Yin et al. 202T), where the adaptation is performed on the physical PDE coefficients only while the NN component is shared across all the environments. GEPS-Phy is our physics-aware model (section 4.3) were all the parameters, PDE coefficients and context vector et are adapted at inference. We also implemented "Phys-Ad", where we use the same formulation as for the hybrid GEPS-Phy, but adaptation is performed on the coefficients of the physical component only while the neural network component is shared across all environments. All the baselines share the same network architectures than the ones used for GEPS, indicated in the implementation paragraph above. More details on basclines implementation are provided in Appendix E.6
In-distribution and dittr---iitribbtion results We report results for and out-of-distribution generalization in table 2 for both the data-driven and the hybrid settings. Across all datasets, our framework performs competitively or better than the baselines for the two settings. Our method is able to correctly adapt to environments in efficient manner, updating only context parameters ca. For the agnostic data-driven experiments, GEPS obtains the best results, although being on the same range of performance as other methods.
The main differentiator of GEPS w.r.t. the baselines lies in lower complexity. In terms of training time, GEPS is way less expensive than gradient- baddeent-basd approaches like CAVIA that involves an outer and inner loop and LEADS, which learns a model specific to each environment. In terms of number of parameters, CoDA needs more training parameters because of its adaptation mechanism relying on a hyper-network. comparison with the baselines terms of parameter complexity is provided in table 10, Additional results in Appendix D show that our data-driven framework adapts faster

---

Table 2: In-distribution and Out-of-distributiio results 32 test trajectories per environ- ment. For out-of-distributiinn generalization, models are fine-tuned on trajectory per environment. Metric is the relative L2 loss. indicates inference has diverged.
Method
Pendulum (x10
Gray-Seott
(x 1x10->>
Burgers (x xx10->
Kolmogorov (x10-)
ln.d
Out-d
In-d
Out-d
In-d
Out-d
In-d
Ou--d
Duta driven
LEADS
20.8 + 1.01
51.1 + 3.47
3.11 0.25
3.81
+ 0.77
6,31 4:0.2
64.1 + 2,65
5.61 0.337
9.18 0.1.1
CAVIA
56.8 1.9.3
91.81 15.8
3.82
23.1 1 6.86
1.06
225 7.94
0.02
8.48 0.16
FOCA
4.30
91.7
1.71 4:110
14.5 3.34
9..2
23.6
6.30:1..02
9.18-10.14
CoDA
21.7 1.08
6612
3.19
2.89
4.03
4,98 0.19
14.5
0.02
6.34 1.11
0.10
3.99
2.22
1. 2.11
2.59
1.94
5.10
Hybrid
APHYNITY
9.66
69.0 1. 0.35
0.04
0.20 1: 0.01
31.9 1.00.8
307 1,40
Phys-Ad
6.14
58.4 # 1.85
1.55 + 1.41
1.82 +: 6.42
15.1 0.10
89.9 + 8.65
GEPS-Phy
8.04
3.64
0.67
0.05
5.43 0.35
3.75
2.78
1.
than the strong baseline CoDA. Concerning the hybrid learning problem, incorporating physical knowledge leads to better results on all datasets except Burgers, where the fully data-driven method performs better. GEPS-Phy better on all the datasets but Gray-Scott for which APHYNITY baseline is best. For Kolmogorov, the baselines did not converged at training.
5.4 Scaling to a larger dataset
To further illustrate the benefits of GEPS, we compare it to CoDA, the SOTA adaptation model, large dataset with a larger model than the ones used for the previous experiments. We use a dataset generated from a PDE with multiple differentiable terms that encompasses several generic PDEs,
inspired from Brandstetter et al. (2022). The PDE writes as (more details Appendix B):
[0,u + 02 (onu ,0_.u + 80 + 70zzz (t, =0
(10)
We report the results in table 3 All the methods trained using ResNet (He al. 2016) architecture, using a context size c of size 8. For In-, we trained our model on 1200 environments with 16 training trajectories per environment and evaluated it 16 new trajectories. For Out-d, we further adapt each model to 10 new environments given context trajectory per environment and then evaluate it 16 new trajectories per environment.
Table 3: In-distribution and Out-distribution results. Metric is the relative L2.
Method
Combined (x10-*
In-d
Out-d
# Params
GEPS
7.52e:3
8.29e
4M
CoDA
9.326-2
1.78e11
35M
GEPS outperforms CoDA in terms of performance and number of parameters. This demonstrates the importance of scalable adaptation for few shot learning: GEPS able to scale on large datasets using deep models while remaining parameter efficient.
6 Discussion and limitations
Limitations We have seen that adaptation essential for learning to generalize neural PDE solvers, and that within this setting, integrating physical knowledge may help. Adaptation still requires sufficient training samples both environments and trajectories. These findings are still to be confirmed for more complex dynamics and real world conditions for which the variety of behaviors should be much larger than for the simple dynamics we experimented with.
Conclusion We empirically demonstrated the importance of adaptation for generalizing to new environments and its superiority with respect to ERM strategies. We proposed new 1s order and low-rank scalable meta-learning framework for learning to generalize time-continuous neural PDE solvers few-shot setting. We also highlighted the benefits of directly embedding PDE solvers as hard constraints in data-dria data-driven models when faced with scarce environments.
10

---

7 Acknowledgments
We acknowledge the financial support provided by DL4CLIM SUIR-21-E223-0017-000 PHLUSIM (ANR-23-CH25-0025-00) and PEPR Sharp (ANR-23-PER) 0008, ANR, FRANCE 2030).
DEEPNUM
References
Akio Arakawa. Computational design for long-term numerical integration of the equations of fluid motion: Two-dimensional incompressible flow. part i. Journal of Computational Physics, 1(I); 119-143, 1966. ISSN 0021-9991. doi: http:///....rg111000100000011000
URL
https:/ http://wwwwwccccccccrrreee.c. com/ wtt.coooccssccc htt.:///eettttcoooocccc//////rta http:///.a.oo//rrr...//////////0
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.
C Basdevant, M Deville, P Haldenwang, J.M Lacroix, J Ouazzani, R Peyret, P Orlandi, and A.T Patera. Spectral and finite difference solutions of the burgers equation. Computers and amp; Fluids, 14(1):23-41, January 1986. ISSN 0045-7930. doi: 10. 10166004--793008)9900666 URL http://dx. dx. doi www.orgg..... 1016/001600000 01116/004-2220 (3806
Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong. Anna Vaughan, Jayesh K. Gupta, Kit Tambiratnam, Alex Archibald, Elizabeth Heider, Max Welling, Richard E. Turner, and Paris Perdikaris. Aurora: A foundation model of the atmosphere, 2024. URL htl.https::. h/ h/ae:://arra //axxiv..ivvvv//aa /rrgi..ov.o.//////
abs abs/2405. 13063
Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data augmentation for neural pde solvers. In International Conference on Machine Learning, pp. 2251-2256. PMLR, 2022.
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. International Conference on Learning Representation, 2023.
Ricky T. Q. Chen. torchdiffeq, 2018. URL htt.://tttt////tttoooooooorrroor
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypemetwork-based meta-learning for
low-rank physics-iffrrmed neural networks. Neural Information Processing Systems, 2023.
Albert Cohen and Ronald Devore. Approximation high-dimensional parametric pdes. Acta Numerica, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2016-December, 2016. ISSN 10636919, doi: 00.1109/CPP..2016.99
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zht -Zu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
Xiang Huang, Zhanhong Ye, Hongsheng Liu, Beiji Shi, Zidong Wang, Kang Yang, Yang Li, Bingya Weng, Min Wang, Haotian Chu, Fan Yu, Bei Hua, Lei Chen, and Bin Dong. Meta-auto-decoder for solving parametric partial differential equations. Neural Information Processing Systems, 2022.
11

---

Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1069-4077. PMIR, 13-15 Apr 2021. URL https://proceediinn mlr
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetorrss. In Annual Meeting of the Association for Computational Linguistics, 2021.
Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonij, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. International Conference on Machine Learning, 2022.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang. Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 2021.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023. URL
http://jmlr www.ooogg..eeeeeee.e wwtgppapprs/v2/// 252244//22122..0..-1 html
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex), 2021.
Zongyi Li. Nikola Borislavov Kovachki, Kamyar Azizzadenesheli. Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric par- tial differential equations. International Conference on Learning Representations, 2021. URL
httpss//opeereeviez htwp:/rwwwwiww.eeeeeeeeeeeeeeeee
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informer neural operator for learning partial differential equations, 2023.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDBs from data. In
Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3208-3216. PMLR, 10-15 Jul 2018. URL httttss:/pproceednnngeeoooo mml.inngsm.........m..mm........ pwwps/evev Vaaass/vv/// 1800/ong18a... html
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. Nat Mach Intell, 3:218 10 2021. doi: 10. 0210884444225--1100000000 URL http:://arxxi...rgg abs/19100 031 www9333tt:tt//// /x doi orgo 10. 1038/ 022////2225202000000000
Junyoung Park, Federico Berto, Arec Jamgochian, Mykel Kochenderfer, and Jinkyoo Park. First-
order context-based adaptation for generalizing to new dynamical systems, 2023. URL https
//openreview perparreriev..nt//frummit.iee http//ffooumm/dddoo000000.a
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- high-performance deep learning library. In Advances in Neural Information Processing Systems Curran Associates, Inc., 2019, URL htt::: seraapperseeurippsee library.
2015-pytorch-- an an--mperrative style- high www.--ighhhhhfooo orw.nce-aaac-ee.. deep- neurips.cc/paper cc/paper/ pdf
32. pp.
John E. Pearson. Complex patterns in a simple system. Science, 1993. doi: 10.1 1611...ccie.....6... 189.
12

---

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 10902716. doi: 10.1016/1.jpp.008.. 10.045.
informed neural networks: A deep learning
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Char- acterizing scaling and transfer behavior. Advances in Neural Information Processing Systems 37 (NeurIPS 2023). 2023.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pffügee, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. Learning neural pde solvers with parameter-guided channel attention. International Conference on Machine Learning, 2023.
Naoya Takeishi and Alexandros Kalousis. Physics-inteerated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems, 2021.
Nilam Tathawadekar, Nguyen Anh Khoa Doan, Camilo F. Silva, and Nils Thuerey. Incomplete to complete multiphysics forecasting hybrid approach for learning unknown phenomena. Data Canta-Centric Engineering, 2023.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference.
Neural Information Processing Systems, 2022.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 2021.
Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long, Transolver: A fast transformer solver for pdes on general geometries. In International Conference on Machine Learning, 2024.
Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting*. Journal of Statistical Mechanics: Theory and Experiment, 2021, 2021. doi: 10.1088/ 1742-5468/ac3ae5.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Neural Information Processing Systems, 2022.
Luisa Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. 36th International Conference on Machine Learning, ICML 2019, 2019-June, 2019.
13

---

A
Related Work
We review data-driven methods for learning parametric PDEs and existing works for incorporating physical priors in neural networks, in the context of dynamical systems.
A.1 Learning parametric PDEs
Traditional ML paradigm The majority of current approaches for learning parametric PDEs considers traditional ERM approach by sampling from the PDE parameter distribution. In MP- PDE (Brandstetter et al.) 2023). PDE parameters are directly embedded in the graph, allowing generalization to PDE parameters sampled from the same data distribution as those used for training. Neural operators (Kovachki et al., 2023 Takamoto et al. 2023) do similarly but require large training sets and often perform poorly for out-o--distribution (OOD) data. Alternatives to ERM such as invariant risk minimization (IRM) have shown stronger generalization by learning domain invariants (Arjovsky et al. 2020; Krueger et al. 2021). However, even for simple classification problems, IRM methods can fail to capture natural invariances and are extremely sensitive to sampling (Kamath et al. 2021).
Multi-environment paradigm Several works have explored multi hattiieeiioommmmmmtt-eeaaannniie settings for dynamical systems. Data-driven methods like DyAd WWngg al. (2022) adapt dynamics models using a time-invariant context from observed state histories, which are often not accessible. Yin et al. (2022) developed LEADS, multi-task method for dynamical systems that adapts in the functional space but requires training a new model for each environment. Park et al. (2023) proposed FOCA address the limitations of second-order optimization of gradient-bases meta-learning approaches (Zintgraf et al. 2019). (Kirchmeyer et al., 2022) make use of a hyper- network for fast adaptation to new dynamics by updating an environment dependent context encoding. While effective, gradient-basee based and hyper- network based approaches respectively involve inner-loop updates or parameter increases with respect to context dimension. Model-based approaches like inffrme- neural networks (PINNs) have been adapted for parametric PDEs (Cho et al.) 2023); Huang et al. (2022) used meta-learning with PINNs through context vectors e learned auto-decoding. One drawback of PINN is their inability to handle scenarios where some physics are unknown.
A.2 Hybrid learning
Data-driven methods with soft constraints Physics losses proposed by (Raissi et al.) 2019) have been used as prior knowledge under the form of soft constraints for neural operators, with the objective to alleviate data scarcity issues (Wang et al. 2021 Li et al. 2023).
Data-driven methods with hard constraints Alternatively, physical priors can incorporated directly hard constraints, leveraging the training of neural networks with differentiable physics. It has been particularly successful for accelerating computational fluid dynamics (CFD) by addressing numerical errors inherent in PDE discretization and for augmenting incomplete physies (Kochkov jtt al. 2021; Yin et al. [021]] Takeishi & Kalousis 2021] TTathaaadeaar et al. 2023). Most these approaches do not consider the parametric PDE setting.
B Dataset details
We present the equations and the data generation settings used for all dynamical systems considered in this work. table 4 we report the different PDE parameters changed to generate training and adaptation environments.
B.1 Damped and driven pendulum equation
We propose to study the damped and driven pendulum equation. The ODE represents the motion of a pendulum which can be subject to damping and forcing term. Even simple dynamical systems such as the pendulum equation can present variety of complicated behaviors (e.g., under-damped, critically damped, resonance, super/sub-hannneee harmonic resonance, etc.), presented in figure
14

---

Table A description of the pivotal factors used to generate environments
PDE
training ODF/DD parameters
adaptation ODE/PD parameters
[0.5, 0.7)
E (0.5, 0.75. 1.0) Wy E (0.3, 0.5, 0.7, F {0.05, 0.15, E (0.1, 0.5)
wy E (0, 0.75, 1.0)
Damped driven pendulum
F E (0.0, 0.1, 0.2)
a E [0., 0.5)
E (Ge- I, be- 5e-4)
E (1.0, 5e-
Burgers
f(a.t) () = [F(sin(wra) cos(wyt)).0)
[(a t) = exp( x²)
wy
w (1.5,300)
E {0.03, 0.039)
F (0.025, 0.042)
2D Gray-Scott
k E {0.058, 0.062]
& (0.050, 0.065]
E [le f(r. (.). = (Fsin(wr)).0)
4]
E (5e- 4)
2D Kolmogorav Flow
(F exp(aryy )) [(cns( ya)) sin(wy()) Domain ([0. 1.25a])
Domain = {10, 0..m). [])
Evvinoment 1
Eavironennt
04-
05.
900
02-
-11
Innineorat 3
Rastmanment
0.5
20-
15
6.5
Ulac Step
Timee
Figure 6: Visualization of different behaviors for the damped and driven pendulum equation
The form of the ODE is:
de d1z
db dt
+ wh sin + a f(1)
(11)
where (() is the angle, wp the natural frequency, a the damping coefficient, [(() is forcing function of the form [(t) = F cos(wy)), where Wy is the forcing frequency and F the amplitude of the forcing. In our case where we suppose we only have incomplete knowledge of the phenomenon, we consider we do not know the damping term. For the initial condition, the pendulum is dropped from a distribution (to) ~ U((0 π 12) and dt
d0 (to) ~ 1).
Data generation Each environment is defined by changes in the system parameters resulting in different behaviors. In training environments, the pendulum either subject to damping forcing, e.g., environment is damped pendulum (F = 0) while environment 2 is a driven pendulum (a = 0). We generate trajectories using a Runge-Kutta solver. For training, we generated distinct environments, each composed of 8 trajectories on the time horizon [0, 25] with a time step At = 0.5. We also generate 32 trajectories on longer time horizon [0, 50) per training environment to evaluate in-distribution generalization. For adaptation, we evaluate our method on 4 distinct environments defined by parameters unseen during training. Only one trajectory is generated per environment with a time horizon is [0, 25) to adapt the model to new dynamics; we evaluate the model's performance on 32 trajectories per environment on time horizon [0, 50). During adaptation, environments are defined such that trajectories are subject both to the forcing and damping term simultaneously.
15

---

B.2
Gray-Scott equation
The PDE describes reaction-diffusion system with complex spatiotemporal pattern through the following 2D PDE:
du D,Au uv² + F(1 - u)
(12)
dt
du D,Au wv² (F + k)v
(13)
di
where u, e represent the concentrations of two chemical components the spatial domain S with periodic boundary conditions. D.. Dr denote the diffusion coefficients respectively for u, v and F, are the reactions parameters. We consider complete knowledge of the physical model, but instead uses RK4 solver with At = 1 instead of using adaptive RK45 solver. We generated environments by changing F, k values, while Du. De are kept constant across all environments. For experiments sampled uniformly values for F and k in ranges [0.03, 0.04] and [0.058, 0.062) respectively. y. For experiments in Section 5.. details about the environments are given table 4
in Section
Data generation We generated trajectories on a temporal grid using an adaptive RK-45 solver. S a 2D space of dimension 32 x 32 with a spatial resolution of As = 2. For training, we generate 4 environments with one trajectory per environment defined on temporal horizon [0, 200]. We evaluate in-distribution generalization on 32 trajectories per environment defined on a temporal horizon [0, 400]. For adaptation, we adapt to 4 new environments in one-shot learning manner. We evaluate the model's performance on 32 new trajectories per environments, defined on a temporal horizon [0, 400).
B.3 Burgers equation
Burgers' equation is a nonlinear equation which models fluid dynamics in ID and features shock formation:
du + du du dt dx dx² + Relooure (ū, (d, f(x,t)
(14)
where u is the velocity field and v is the diffusion coefficient and f(z, t) is a forcing term. The unresolved scales Relosure (ū, u) and the forcing function is typically unknown, and needs to be directly learned from data. For the experiments done in Section 3 environments have been generated by sampling uniformly v E [1e 4, 0.5). For experiments in Section 5.3 we generated environments by changing the diffusion coefficient or changing the forcing function, as detailed table
Data generation For the DNS, we generate complex trajectories using 5th order central difference scheme using Runge-Kutta 45 solver with time-step Al 1e-5 and Ar are particularly costly generate, therefore, we rather use LES. To obtain the ground truth LES trajectories, we apply a spatial filtering operator on the DNS trajectories. We also down-sample the temporal and spatial grid. Therefore, we obtain LES trajectories with timestep At 1 and Az 256
16384 Such trajectories
2
For training, we generate 6 environments with 4 trajectories per environment. Trajectories have been generated on a temporal horizon [0, 0.05) for training and for in-distribution evaluation. For adaptation, we adapt our model on 4 new unseen environments using only trajectory per environment. We evaluate out-distribution performance on 32 new ICs on time horizon [0, 0.05). For the generation of the data, we define [0, 2T] periodic domain and consider the following initial condition:
2
k
1 kin
A: Ko
E(k) - 35
exp
(15)
Ain
Velocity is linked to energy by the following equation
u(k) (2k
(16)
B.4
Kolmogorov flow
We propose a turbulence equation. We focus on analyzing the dynamics of the vorticity variable. The vorticity, denoted by w, is vector field that characterizes the local rotation of fluid elements,
16

---

defined as w V x u. Like Burgers, we generate LES, leading to an unknown unclosed term appearing in the vorticity equation, expressed as:
Ou Ōt + (u VJw - u + Ru, = f(x,y,
(17)
Here, u represents the fluid velocity field, the kinematic viscosity with V/ R/ Re and f(z y, t) is a forcing term. The unresolved scales Relosure (ū, u) and the forcing function is typically unknown, and needs be directly learned from data. For the vorticity equation, environments can be defined either by changes in the viscosity term, the domain size or the forcing function term.
Data generation For the data generation of DNS, we use a point stencil for the classical central difference scheme of the Laplacian operator. For the Jacobian, we use second order accurate scheme proposed by Arakawa that preserves the energy, enstrophy and skew symmetry (Arakawal 1966). Finally for solving the Poisson equation, we use Fast Fourier Transform based solver. We discretize a periodic domain into 512 x 512 points for the DNS and uses a RK4 solver with At obtain the ground truth LES by applying spatial filtering operator on the DNS trajectories. We also wammle temporal and spatial grid, thus obtaining LES trajectories with timestep At on a 64 x 64 grid.
3. We
We generate environments with 16 trajectories per environments for training. Trajectories have been generated on a temporal horizon [0, 1] for training and [0, 2] for in-distribution evaluation. During adaptation, we adapt our model on unseen environments in one- shot manner, using only one trajectory per environment. We evaluate out-distribution performance on 32 new ICs on a time horizon [0, 2]. We consider the following initial conditions:
4 V E(k)
h:
(-(
(-
A:
exp
(18)
3
ko
Ku
Vorticity is linked to energy by the following equation:
E(k)
w(k)
(19)
kk
B.5
Combined equation
We used the setting introduced in |Bandstetter et al. (2023). but with the exception that we do include forcing term and add the orddr spatial derivative. The combined equation is thus described by the following PDE:
1ā,u Ox(2²² -30,u
+ 50 + 7dzz [t)
(20)
uo(x)= = Σ Aisiin (iTTn(22 b.))
(21)
j=1
a, a,3,6 8 and y are the parameters that are varied.
Data generation We used the spectral solver proposed Brandstetter et (2022) to generate the solution. We sampled 1200 environments for training, by sampling uniformly within the ranges E [0.5, 1), 8 e [0, 0.5). 5 E [0, 1] and y [0, 1]. For each parameter instance, we sampled 16 trajectories, resulting in 19200 trajectories. We then evaluate it on 19200 new trajectories. The trajectories were generated with a spatial resolution of 256 on a temporal horizon [0, 30). We only keep 10 time-steps. For out-distribution evaluation, we sample 10 new environments from the same ranges of parameters, but for parameters not seen during training. We have access one trajectory from each evaluation environment for adapting the model, and evaluate it on 16 new trajectories.
C Trajectory based formulation
In practice, F unavailable and we can only approximate from discretized trajectories. As done in (Kirchmeyer et al. 2022). we use trajectory-basee formulation of Eq. (9). We consider set of
17

---

trajectories discretized over a uniform temporal and spatial grid includes T. ( )d states, where d, the spatial dimension. At and As represent respectively the temporal and Ae S are the temporal horizon and spatial grid size. Our loss writes as:
spatial resolution. T and
N T//A (() = Σ
(s (5
(22)
where 7 (4) u + f (o(d)
"5(sks 4) is the state value in the 5th trajectory from environment the spatial coordinate 8 and time 4 A Iât. (t) & [u(s2, i)..................... (/ t)]² is the state vector in the jth trajectory from environment e over the spatial domain time and uh the corresponding IC.
D Additional results
We conduct comprehensive ablation study across various datasets assess the robustness of our method. Our investigation explores several key aspects, including the learnability of PDE parameters within our hybrid framework. We also examine the adaptability of context-based methods new, unseen environments, with varying numbers of adaptation trajectories Naa- Then, we show the effectiveness of our model in terms of complexity and performance when varying the size of the context vector. Additionally, we study the impact of initialization strategies for adaptation parameters in our meta-learning framework. Finally, we demonstrate the ability of our method adapt rapidly to unseen environments with minimal gradient updates.
D.1 PDE parameter estimation
Existing works combining hard physical constraints and data-driven components either learn the PDE parameters or assume availability of the real parameters for single PDE instance (Yin et al. 2021). In our case, environment. In section 4.3 we proposed two strategies for estimating the PDE parameters. We report in the Figure the training and adaptation MAE of the PDE parameters pendulum equation, where the first strategy has been adopted. In Figure TO we report the training and adaptation convergence for the gray-scott equation, where the second strategy has been applied.
trying to learn the PDE parameters in unsupervised manner for each
Figure MAE loss of the PDE parameters for the Pendulum equation during training and adaptation
Convergeaze of PDE Pernester Estreation
Convergenco PDE Pururater Estreation
67"
4.000
4008
4008
Epwers
laxa
Figure 8: Convergence of PDE parameter esti- mation during training
Figure 9: Convergence of PDE parameter esti- mation during adaptation
Our framework successfully estimates PDE parameters during both training and adaptation steps for the Pendulum and Scott equations. Notably, the accuracy of our framework in predicting PDE parameters is heavily dependent on the initialization of physical parameters. The specific initialization values used for PDE parameters are detailed in Table E.
18

---

Figure 10: MAE Convergence of the PDE parameters for the Gray-Scott equation
Converpeace
Convergeaco
UAA Proerate decie
VMA Poomeeke Enience
Figure 11:
Convergence of PDE parameter esti-
Figure 12: Convergence of PDE parameter esti-
mation during training
mation during adaptation
D.2 Number of adaptation trajectories
Context adaptation While our methods remain effective on the Gray-Scott and the Burgers equation when adapting to unseen environments, meta-learning frameworks perform poorly the Kolmogorov flow equations. This particularly due to (i) the complex dynamics and very different behaviors that appear, (ii) the diversity of behaviors observed within same environment for different initial conditions. Thus, the number of trajectories for training and for adaptation need to be sufficiently large for learning such dynamics; one-shot learning is therefore very complex task for such systems. We report in table the results when increasing the number of adaptation trajectories for Kolmogorov flow when adapting only context vectors:
Table Relative MSE loss with respect to number of adaptation trajectories when adapting context vectors ca.
Number of adaptation trajectories Nai
Model
1
8
16
CoDA
6.23e-1
6,22e-1
6.22e-1
6.20e-1 1
CAVIA
6.42e-1
6.40e-1
6.33e-1
6.29e-1
GEPS
5.22e-1
5.22e-1
5.21e-1
5.20e-
We remark that context based frameworks do not scale well when increasing the number of adaptation trajectories.
Low-rank adaptation To overcome this issue, we propose to take advantage of the low-rank adaptation formulation proposed, which enable cost-efficient adaptation of our model with respect to adapting all the parameters of the model. Instead of adapting c², we adapt parameters Ac B. We report results in table C
Table Relative MSE loss with respect to number of adaptation trajectories when adapting low rank adaptation parameters.
Number of adaptation trajectories Na
Model
1
8
16
GEPS
4.5e-1
4.23e-1
3.97e-1
3.64e-
D.3 Data-driven Parameter initializations
Initialization of adaptation parameters essential for correctly adapting to environments. We study how initialization impacts the training of our model the Gray-Scott and Burgers equation. In LoRa,
19

---

it is typically advised to initialize A using a Gaussian distribution and set B = 0. In table 7 we report the relative loss for different initialization, including LoRa initialization. On both Gray-Scott and Burgers equation, the orthogonal initialization performs best.
Table 7: Relative MSE loss with respect to parameter initialization
Model initialization
Gray-Scott
Burgers
Kaiming Xavier
7.26e-2
1.50c-2
NaN
1.57e-2
LoRA init
2.35e-1
4.95e-1
Orthogonal
2.22c-2
1.46e-2
D.3.1 Code dimension
In CoDa, the authors advocate choose relatively small code size, corresponding to the degree of variations defining each environment. In our case, we proposed datasets with more degree of variations than the datasets used by (Kirchmeyer et al. 2022), presenting much more differences, necessitating larger code dimension size. In table 8] we show the performance of our model on both
Gray-Scott and Burgers equation when varying the code dimension for CoDA and method and the increase in number parameters.
Figure 13: Model size with respect to code dimension
GDD
1200K
53P%
1305
Modal Sta fer Gayy Senti POR
1000
300
600%
100K
200
Dimersoo of the Contest
Table 8: Relative MSE the full trajectory with varying code dimen-
sion
CoDA
GEPS
code size |
Burgers
Gray-Scott
Burgers
Gray-Scott
2.04e-2
9.72e-2
1.49e-2
1.10e-1
2
2.03e-2
8.73e-2
1.35e-2
7.12e-2
4
1.96e-2
9.29e-2
1.46e-2
5.65e-2
8
1.78e-2
9.38e-2
1.43e-2
7.27e-2
16
1.84e-2
1.0le-1
1.30e-2
5.82e-2
D.3.2 Adaptation speed
One important property of meta-learning framework their ability to adapt to unseen environments in few adaptation steps. Our framework allows fast adaptation to new environments, compared to state-of the method like CoDA. We report in Figure the convergence of the loss during adaption for both our method and CoDA. With our framework, we able to adapt to new environments in
20

---

Our
CoDA
0.14
0.12
Relative MSE
0.10
0.02
0.00
200
400
600
800
1000
Epoch
Figure 14: Convergence speed of the model Go adapt new environments for the Burgers dataset
less than 100 steps, compared to CoDA which needs 500 steps. For both runs, we used the same learning rate 0.01.
D.3.3 Out-range temporal horizon generalization
We evaluate the ability of the different adaptation mechanisms for out-range temporal horizon extrapolation. All models have been trained on temporal horizon [0, 7". We thus evaluate the performance of our models when extrapolating outside the trained temporal horizon. For out-range extrapolation, models are evaluated on the horizon [T. 2T]. We report in-distribution and out- distribution performance in table Error accumulates over time and leads lower performance outside the training horizon, but GEPS still outperforms existing baselines.
Table 9: Generalization results for in-distribution and out-distribution environments Test results for out-range time horizon (Out-t), Metric is the relative L2 loss. indicates inference has diverged.
Dataset
Pendulum
Gray-Scott
Burgers
Kolmo
Type
Model
In-d
Out-d
In-d
Our-d
In-d
Out-d
In-d
Out-d
LEADS
6.30e-1
7.28e-1
1.45e-1
1.27e-1
4.55e-2
3.22e-
9.80e-1
1.01
CAVIA
40.4
1.88
3.73c-1
5.51c-1
6.44c-2
6.72c-1
1.06
1.06
FOCA
14.0
1.08
3.56c-1
3.34e-1
3.12c-1
4.60c-1
1.0
1.05
CoDA
9.54c-1
9,75c-1
1.52c-1
1.19c-1
3.41c-2
3.35c-
8.29c-1
9,99c-1
Data-driven
GEPS
4.92e-1
8.45e-1
8.37c-2
6.91e-2
2.23e-2
2.68e-1
7.70c-1
9,74c-1
APHYNITY
1.32
6.05e-3
4.19e-3
1.12e-1
8.09e-1
Hybrid
GEPS-Phy
2.88e-1
7.12e-1
2.38e-2
1.92e-2
4.72e-2
3,59e-1
7.33e-1
9.32e-1
D.3.4 Training time and number of parameters
report table TO the training time and number of parameters for each adaptive conditioning method for all datascts:
21

---

Table 10: Number of parameters (# Params) and training time (Time) for all different adaptive conditioning methods.
Method
Pendulum
Gray-Scott
Burgers
Kolmogorov
# Params
Time
# Params
Time
# Params
Time
# Params
Time
LEADS
43K
10h
380K
14h
400K
1d 19h
1.8M
2d 23h
CAVIA
4h
76K
1d 15h
58K
4d 19h
278K
4d 21h
CoDA
43K
6h
380K
5h
280K
10h
726K
2d 12h
GEPS
13K
5h
90K
4h
69K
9h
387K
2d 2h
E
Implementation details
The code has been written in Pytorch (Paszke al. 2019). All experiments were conducted single GPU:NVIDIA RTX A5000 (25 Go). We estimate the computation time needed for development and the different experiments to approximately 200 days.
E.1
Architecture
Adaptation
Wbe
Aidiag(c +
(())
(02
xt(
Pp
W,
(t)
8p (05)
(.g(0).c
1.(of ((0).ct)
(1)=f-1(0).c")
ODESOLVE
Gp
Ga
u (t
Our adaptation framework for the two instances of our model: agnostic and physics-aware. Blocks in blue refer to the physical model Ip and data-driven component Ja. Blocks in pink refer to the trainable modules. The green block describes the adaptation mechanism for the physical and data-driven components. We use NeuralODE (ODESOLVE) for time-integration.
Figure 15:
E.2 Data-driven model
We implement the dynamical model Go with the following architectures:
Pendulum:
we used MLPs composed of 4 hidden layers of width 64.
Gray-Scott: 4-layer 2D ConvNet with 64-channel hidden layers, and 3 convolution kernels with circular padding.
Burgers:
4-layer ID ConvNet with 64-channel hidden layers, and x 7 convolution kernlss with circular padding.
Kolmogorov flow: Fourier layers with 12 modes and a width of 16.
For all networks, we use Swish activation layers. We an RK4 solver for the Neural ODE.
22

---

E.3
Physical model
For each dataset, we assume availability some prior knowledge of the studied dynamical system. given as hard constraints. We report in the table the physical knowledge assumed, the number of training parameters and the values of the parameters used initialization:
Table 11: Physical model details
Physical Model
# trainable params
Parameter initialization
Pendulum
dgg dea +
w sin
F cos(wyt)
3
w
0.1, F 0.2, wy 0.5
Burgers
du
v
d2u
1e
dt
de
du 3 df
D
+ ur²
F(1 uu)
5e 2 5e
Gray-Scott
D, + uo² + (F + k)u
Kolmogorov
(u V)w - uV2w
5e
E.4 Optimizer
For all datasets, we the Adam optimizer and (31, 32) (0.9, 0.999) for both training and adaptation. For training, we used a learning rate scheduler which reduces the learning rate when the loss has stopped improving. We set the threshold 0.01 with a patience of 250 epochs with respect to the training loss. The minimum learning rate is le 5.
E.5 Haper-parameter details
We report the hyper- pepeee-prametees used for each dataset in the table:
Table 12: Framework sypem-parameters
Hyper-parameter
Pendulum
Burgers
Gray-Scort
Kolmogorov Flow
16
4
4
4
depth
4
4
4
4
Go
width
64
64
64
64
activation
Swish
Swish
Swish
Swish
batch size
8
4
4
4
epochs
20000
20000
20000
20000
Training paraeenarameters
learning
1e-2
1c-2
1c-2
1c-2
Scheduler decay
0.9
Teacher forcing
No
No
No
batch size
4
4
4
Adaptation saper-parameten
epochs
500
500
500
500
learning rate
le-2
le-2
le-2
1e-2
E.6
Baselines implementation
For all baselines, we followed the recommendations given by the authors.
ERM experiments
We use the following baselines for the ERM experiments:
MP-PDE:
We implement MP-PDE as a 1-step solver, where the time-bundling and pushfor- ward trick do apply. We use 6 message-passing blocks and 64 hidden features. We build the graph with the 8 closest nodes. We use learning rate of le - and a batch size of 16. We train for 5000 epochs on Burgers and Gray-Scott equations.
FNO: FNO is trained for 5, 000 epochs on Burgers and Gray-Scott equations with learning rate of 1e-3. We used 12 modes and a width of 32 and Fourier layers. We also use step scheduler every 250 epochs with decay of 0.5.
23

---

We implement a CNN as 1 1-step solver, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scotr equations with a learning rate of le 3. We also use a step scheduler every 250 epochs with a decay of 0.5.
CNN:
GEPS: We implement CNN 1-step solver with multi-environment setting, composed of 4 layers of 64 features with Swish activation functions. We train our model on 5000 epochs on Burgers and Gray-Scott equations with a learning rate of 1e We also use step scheduler every 250 epochs with decay of 0.5.
In-distribution and out-distribution experiments ization experiments:
We use the following baselines for the general-
LEADS: We implement one-per-env method of the multi-task learning framework LEADS.
For all datasets, we train the model during 20000 epochs with learning rate Ir = 0.01, with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. All other hyper-parameters are the same than the one used in the paper.
CAVIA: We adapt CAVIA's framework for learning dynamical systems. We train CAVIA during 20000 epochs with a inner learning rate Iriner 0.1 and an outer learning rate Iouueer 0.001. For all datasets, we train the model during 20000 epochs with a step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets. We tested different inner step values, and fixed to 7 for all datasets. We followed the authors recommendations for the experiments.
FOCA: We implement first-order gradient- based method called FOCA to learn dynamical systems. We train CAVIA during 20000 epochs with an inner learning rate Irinner = 1.0 and outer learning rate Iroeer 0.001. For all datasets, a step scheduler every 250 epochs. We used T = 1 for the exponential moving average and 5 inner steps.
train the model during 20000 with
CODA: We implement CoDA, meta-learning framework for learning dynamical systems. We implemented the 11 and 12 regularization of the CoDA framework. We train the model during 20000 epochs with step scheduler every 250 epochs. For the size of the contexts, we first fixed the size following the authors recommendations but we observed improved performance using the same dimension used with framework.
We implemented multi-environnen version of the APHYNITY framework. We learn physics for each environment and learn to correct the physics model using a data- driven model that corrects all environments. During adaptation, the model. For all datasets, we train the model during 20000 epochs with a learning rate Ir 0.001, with step scheduler every 250 epochs. We used the same batch size used for our framework across all datasets.
APHYNITY:
tune all the parameters
F Qualitative results
In this section, show different visualization of the predictions made by our framework and compare with the different baselines.
F1
In-distribution and out-distribution generalization for Burgers dataset
We provide in Figure 16 in-distribution and out-distribution generalization for the Burgers equation.
F22 Out-distribution generalization for Gray-Scott dataset
We provide in Figure 17 disttribibtn generalization for the Gray-scott equation.
24

---

F.3
In-distribution generalization for Kolmogorov dataset
We provide in Figure 17 in-distribution generalization for the Kolmogorov equation.
Model
In- In-Domain gencralisation
Out-Domain generalisation
Ground truth
GEPS
CoDA
CAVIA
LEADS
Figure 16:
Comparison of in in-distribution and out-distribution predictions of trajectory on ID
Burgers. The trajectories are predicted from = 0 (purple) to = 0.1 (red).
25

---

Model
Initial condition
Predicted trajectory +
In-t
Out-t
Ground truth
O
GEPS
CoDA
O
CAVIA
O
LEADS
Figure 17: Prediction per frame for our approach on 2D Gray-Scott for an oftto--distiibttion trajectory. The trajectory is predicted from t= 0 to == T'. In our setting, T= 19 and T 39.
26

---

Model
Initial condition
Predicted trajectory
In-t
to
i
Ground
truth
GEPS
CoDA
CAVIA
LEADS
Figure 18: Prediction per frame for our approach on 2D Kolmogorov flow for an in-distribution trajectory. The trajectory predicted from 0 to T'. our setting, T 19 and T 39.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provided experimental results in section 3 putting in evidence the impor- tance of adaptation mechanisms for learning parametric PDEs, while ERM tend fail in IVP. Despite working when considering a history, we show our method still outperforms such methods. We then propose data-driven and physics-aware meta-learning framework for learning to generalize on unseen environments, showing strong performance compared to existing meta-learning works in 5.3 Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We adressed the limitations of our framework in the section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results are provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided details Appendix E about the seper-parameters used train our model, including model architecture. We provided details for both experiments done in section 3 and [.3 We also provided how baselines were trained for both experiments. We
also included pseudo-code for our meta-learning framework, in algorithm
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
29

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provided open access to our code and access to the code used to generate our data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips.com ptappssccppbblic/uuiieeeee http://guuiieeeeeeee wwwiideeeeeooSSSmmeeeeeeee spliccoicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimiser, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided in section all details to generate the different environments for our datasets and the number of training and test trajectories generated, for both training and adaptation settings. We also provided in all details concerning the choice of optimiser and learning schedulers. We provided in Table the hyper-parameter used for each dataset.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: did provided error bars to show statistical significance of our experiments for the second set of experiments in section 5.3 We did not include it for section 3, this is
particularly due to the large number of experiments done and the number of baselines used, which require extensive computational demand.
30

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided in the computer resources used for our experiments with the memory of the GPU used. We also provided time of execution associated for the main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///eerrrr..eeeellee/ee/////
Answer: [Yes]
Justification: We do not use personal data and credits in the code will be properly mentioned when needed. We also provided all config files for reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
31

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work is applied to physical sciences with generated data, so the appli- cations are not related to socictal concerns such disinformation or privacy and security considcrations.
Guidelines: Our work is applied to physical sciences, so the applications are not related to societal concerns such as disinformation or privacy and security considerations.
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We used synthetic data, not intended for widespread use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
32

---

Justification: We credited the corresponding authors for the packages, baselines and datasets used in the experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprssii thcode com datasatss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We released our code GitHub.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines: We not crowdsourcing.
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
33

---

Answer:
[NA]
Justification:
Not relevant here.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "BW69ulOgLa", "reviewer": "Reviewer_3TLD", "summary": "The authors propose GEPS, a method for meta-learning neural solvers for parametric PDEs. Similarly to gradient-based meta-learning methods such as MAML, the authors formulate the problem as a two-step optimization problem: the goal of the first step is to learn a model that's able to easily adapt to new environments via learning context-dependent parameters in the second step. A data-driven and physics-based decomposition is also proposed. The authors evaluate on four canonical dynamical systems and demonstrate state-of-the-art performance.", "questions": "- Why is it that for Burgers, the hybrid setting of GEPS performs worse than the purely data-driven setting? Does this suggest that the data-driven/physics-based decomposition (Eqn 6) is not expressive enough to handle certain PDEs?\n- In Section 3.3.3, two strategies for adapting PDE parameters to a new environment are proposed (lines 213-219): learning $c^e$ only or learning both $c^e$ and $\\theta^e_p$. Which strategy is used for Tables 2 and 3?", "limitations": "The authors address the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and clear.\n- The proposed method is straightforward and effective. The low-rank parameterization applied to meta-learning for dynamical systems is novel, as far as I am aware.\n- Experiments are thorough and comparisons to other meta-learning models are detailed. Evaluated on four canonical dynamical systems, the results are compelling.", "weaknesses": "- More experiments investigating the physics-aware component of the model would be illluminating. It's mentioned in Section 3.3.3 for the method to incorporate incomplete physics information, e.g. incomplete set of terms or inexact coefficients. However, evaluations of the hybrid method are only done with full knowledge of the system coefficients.\n- More ablation studies could be useful to further clarify the importance of each component of the proposed model. For example, it's unclear to what extent the improved performance is due to the specific low-rank parameterization (Eqn 3).\n- It would also be helpful to include the number of parameters of all methods evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and have addressed the raised concerns below.\n\n### More experiments investigating the physics-aware component of the model would be illuminating:\n\nFor the hybrid physics-ML setting, we make two assumptions:\n- the physics is only partly known and shall be completed by a NN component for the targeted forecasting task; this is the forward problem. For the pendulum, the physics component ignores the damping term,  for Burgers and Kolmogorov, the model ignores both the forcing and the LES terms. The only exception is the Gray-Scott example, for which one assumes full knowledge of the equation, but the physical parameters are unknown and shall be estimated (see below).\n\n- for all datasets, the parameters of the partial physics component are unknown and shall be estimated - this is the inverse problem. The list of parameters to be estimated is provided in appendix D.3 - table 9. For example for the pendulum, 3 parameters $(\\omega_0, F, w_f)$ are estimated. Figures 7 to 11, appendix C1, show the error (MAE) for the estimation of the parameters for pendulum and Gray-Scott (this is an average over all the parameters - 3 for the pendulum, 2 for GS).\n\n\n### More ablation studies could be useful to further clarify the importance of each component of the proposed model:\n\nWe report 4 different ablations that could justify the importance of our low-rank module.\n\nWe have added a new baseline denoted **Phys-adaptor**, where we only adapt the physical component to new environments while the data-driven component is shared across all environments, without (low-rank) adaptation. The results, indicated on tables 3 and 4 on the pdf file - row **Phys-adaptor**, show that this model performs worse over all the datasets compared to GEPS, highlighting the importance of adaptation for the NN component.\n\nOther ablations appear in the appendix.\n- in section C.2, we evaluated different context-based meta-learning frameworks (CAVIA, CODA, GEPS) during adaptation while increasing the number of samples per environment. We demonstrated that low-rank adaptation performs better than the alternatives for complex PDEs (Kolmogorov). For example, its performance improves with the number of training trajectories, whereas existing methods quickly reach a plateau and do not show further improvement.\n- in section C.3.1, we showed that our framework scales better than reference method CoDA in terms of numbers parameters and performance when varying the size of the context.\n- Figure 13, appendix C.3.2, shows that our method allows faster convergence during adaptation compared to reference CoDA.\n\n### It would also be helpful to include the number of parameters of all methods evaluated.\n\nWe reported in the PDF rebuttal page (table 1) the training time and the number of parameters used for each baseline for 1D (Burgers) and 2D (Gray-Scott) PDEs. \n\n### Why is it that for Burgers, the hybrid setting of GEPS performs worse than the purely data-driven setting?\n\nWe were also surprised by this results, and redo the experiments. We have no clear explanation, but we believe that the high flexibility of our formulation for combining $g_p$ and $g_a$ can make training complex for some PDEs and causes over-fitting, especially considering the low-amount of data considered in our setting.\n\n### Two strategies for adapting PDE parameters to a new environment are proposed. Which strategy is used for Tables 2 and 3 in the paper?\n\nFor Pendulum and the Burgers, we used the context $c^e$ to learn the physical parameters. For Gray-Scott and the Kolmogorov, we directly learned the physical parameters without using the context vectors. The two strategies provide similar results, we reported in the paper the best results among the two alternatives.\n\nWe hope we answered the main concerns which will be made clearer in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0XgPayx60r", "reviewer": "Reviewer_JETf", "summary": "In this paper, we propose a meta-learning method called GEPS, which utilizes an adaptation approach to generalize a PDE solver to unseen environments. This method demonstrates better generalization compared to classical ERM approaches.", "questions": "* The overall relative error scale of FNO and MPPDE in Figures 2 and 3 and Table 1 is around 1e-1 (even when the environment and trajectory are sufficient), which appears to be generally larger than the errors presented in the respective original papers. Is this due to the difficulty of the equations, or is there another reason for the relatively large errors?\n\n* I am not quite clear on how the physical knowledge insertion $g_p$ and the data augmentation $g_a$ in Equation (6) are each encoded. Could you provide a more detailed explanation of the encoding methods?\n\n* What is the significance of the order of $g_p$ and $g_a$? Would it cause problems if $g_a$ were applied before $g_p$?", "limitations": "See above comments.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This model can adapt to a new environment \\( f^e \\) in one shot and predict the time trajectory of the corresponding PDE at inference time. Specifically, the proposed model outperforms other existing meta-learning methods both in in-domain and out-domain environments of the initial conditions, as well as in in-range and out-of-range (extrapolation) time horizons.", "weaknesses": "Section 3.2 provides the motivation for the proposed method, but I am curious about how this motivation relates to the methods proposed in Sections 3.3.2 and 3.3.3. It would be helpful to explain how the issues identified with existing methods in the motivation section led to the use of neural ODEs and the model structure presented in Figure 4. Additionally, considering the results in Tables 2 and 3, the improvement in error scale compared to existing meta-learning methods does not seem significant. Figures 14 and 15 also show minimal differences. Therefore, it would be beneficial to experimentally demonstrate other advantages or unique aspects of the proposed GEPS method beyond its accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're thankful for the reviewer's helpful feedback and have addressed the raised concerns below.\n\n### Section 3.2 provides the motivation for the proposed method, but I am curious about how this motivation relates to the methods proposed in Sections 3.3.2 and 3.3.3:\n\nThe main messages are:\n\n- Classical ERM training is inadequate for modeling the variability in dynamical physical processes (framed as parametric PDEs), as shown in section 3.2.\n- An alternative inductive principle is needed. We propose an adaptive conditioning principle: learning from multiple environments to rapidly adapt to a new one (section 3.3). The key focus is developing an effective meta-learning strategy.\n- We propose an implementation that is computationally efficient (fast adaptation to new environments) and scalable w.r.t. the number of training samples via low-rank (3.3.1) adaptation and first-order optimization (3.3.2). This framework is adaptable to different NN backbones. Experiments show it works well with MLP, CNN, and FNO.\n\nTo further assess the generality and applicability of our framework, we consider two settings:\n- Pure data-driven approaches with no prior physics, where everything is learned from data.\n- Hybrid formulations with a PDE prior that partially describes the underlying phenomenon (section 3.3.3).\n\nThese settings correspond to common real-world situations explored in the literature.\n\n**Use of Neural ODE**: \"Neural ODE\" refers to a family of time integration methods. We use a popular RK4 solver, but other solvers could also be used. This is only a relevant technical choice, e.g., for the ERM methods, we used the same time-stepping method proposed in the corresponding papers (FNO, MPPDE, Transolver, CNN) for GEPS.\n\n### The improvement in error scale compared to existing meta-learning methods does not seem significant. Figures 14 and 15 also show minimal differences.\n\nWe have now provided error bars for in-domain and out-domain experiments (tables 3 and 4 in the pdf companion), highlighting that the results obtained are consistent and significant for most cases. Regarding qualitative results, all the methods are able to reconstruct Burgers trajectory (Fig 14). For Gray-Scott (Fig 15), GEPS clearly outperforms CAVIA and LEADS. Only CoDA matches GEPS, but diverges from the ground-truth for out-range time horizon. For the Kolmogorov dataset (Fig 16), only GEPS is able to predict accurately the trajectory, and starts to diverge from ground truth for out range time horizon.\n\n### Advantages or unique aspects of the proposed GEPS method beyond its accuracy.\n\n- We provided training times and number of parameters of GEPS compared to other methods in the PDF rebuttal page (table 1): GEPS is faster to train and requires less parameters than concurrent approaches.\n- in appendix C.2, we showed that unlike other methods, GEPS performance increase with the number of adaptation trajectories while other methods performance quickly saturate.\n- in appendix C.3.1, we showed that GEPS achieves greater performance and scales better in terms of trainable parameters compared to reference CoDA when increasing the context vector size. \n- In C.3.2, we experimentally observed that GEPS can adapt to new environments faster than CoDA.\n \n### The overall relative error scale of FNO and MPPDE in Figures 2 and 3 and Table 1 is around 1e-1 which appears to be generally larger than the errors presented in the respective original papers. Is this due to the difficulty of the equations, or is there another reason for the relatively large errors?\n\nThe error of FNO and MP-PDE is higher because the datasets used in the respective papers have been generated by a single equation with fixed parameter values while we consider multiple environments (i.e. a distribution of parameters).\nWhatever the backbone, including FNO, MP-PDE and the new Transolver (see additional experiments below), the ERM induction principle is not adequate for learning from multiple physics environments when only an initial condition is given (Initial value problems). \n\nNote that we have added a SOTA Transformer neural operator (NO) the **Transolver**, to the experimental comparison (see Fig. 1 and table 2 in the in the pdf companion), with the same conclusions.\n\n### How the physical knowledge $g_p$ and the data augmentation $g_a$ in Equation (6) are each encoded?\n\nThe physical component of the model is encoded as a differentiable numerical solver. Being differentiable, the latter can be combined with any NN components and the whole architecture trained end to end. As for the combination in equation 6, given an initial condition $u_0$, the physical model $g_p$ takes as input $u_0$ and outputs an estimate of the temporal derivative $du/dt$, which is processed by the data driven term $g_a$. The latter then complements the incomplete estimate computed by the physical component.\n\n### What is the significance of the order of $g_p$ and $g_a$ Would it cause problems if $g_a$ were applied before $g_p$\n\nDifferent approaches have been considered in the literature. [1] considered that the data term and the physical term should be combined in an additive manner: $g = g_a + g_p$. In our case, we think that $g = g_a \\circ g_p$ is a more general formulation. As for the order, since one makes the assumption that $g_p$ can be incomplete, it is natural to consider that $g_a$ augments this incomplete knowledge and operates after the physical component.\n\n[1] Yin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, ICLR 2021\n\nWe hope it answers your remarks. We will make sure that yours concerns are made clearer in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eg5BFw2Ekp", "reviewer": "Reviewer_m65H", "summary": "This paper focuses on the PDE solver generalization and proposes the GEPS model based on a low-rank-based meta-learning strategy. Specifically, the authors config the PDE solver as a low-rank framework, where different environments correspond to domain-specific diagonal matrices. During adaption, only the diagonal matrices are trained. Further, they employ a hybrid framework for incorporating physics prior as a trainable module in the model beginning. The authors provide extensive in- and out-domain experiments and compare the model with diverse baselines.", "questions": "About the ERM experiments, will the model know the correct PDE coefficient during training and adaption? Do these experimented ERM baselines use the same information as GEPS?", "limitations": "They have discussed limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tThis paper focuses on an important problem, which is PDE generalization.\n-\tThe proposed method is reasonable and performs well in most cases.\n-\tThis paper is well-written and clear.", "weaknesses": "1.\tThe technical contribution is quite limited.\n\nUtilizing shared backbone and domain-specific weights for multi-environment or multi-task learning is a widely used design. There are many adaptor-based models, such as [1]. As for the hybrid framework, I think the authors fail to elaborate on the motivation of the overall design. Why use the prior module at the beginning? What about using it at the end of the model or using it as an adaptor? Also, why does the physics module not use the same low-rank design as the data part?\n\n[1] Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021\n\nIn addition, fine-tuning the diagonal matrix is also related to SVD-PINN [2], which should also be included as a baseline.\n\n[2] Svd-pinns: Transfer learning of physics-informed neural networks via singular value decomposition, SSCI 2022\n\n2.\tAbout experiments\n\nSome simple but important baselines are missing, including the adaptor-based method and some generalizable backbones. Specifically, for the adaptor-based method, the authors can train different adaptors for different environments based on a shared backbone. As for the generalizable backbones, they can experiment with advanced neural operators, such as [1,2]. I mean that train a universal model with thes backbones (the PDE coefficients are concatenated with inputs) and directly generalize them to new PDEs or coefficients. These baselines should also be included in Section 3.2 experiments.\n\n[1] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024\n\n[2] Scalable Transformer for PDE surrogate modeling, NeurIPS 2023\n\nAs a supplement to the main results in Tables 2 and 3, the training time of different methods should be compared.\n\n3.\tComparison with ensemble in Figure 3.\n\nI think ensemble could be a strong baseline, where it performs closely to GEPS in Burgers. Maybe the authors should replace the backbone with more advanced Transformer-based methods.\n\n4.\tAbout the title “adaptive conditioning”.\n\nThis paper proposed a low-rank training strategy. Thus, I think the adaptive conditioning is unsuitable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and recommendations to improve the paper. We have addressed the raised concerns below, including additional experiments as suggested.\n\n### Utilizing shared backbone and domain-specific weights for multi-environment or multi-task learning is a widely used design.\n\nMaybe the paper was not clear enough, but the main focus is on the general adaptation framework, not the specific implementation. The main messages are as follows:\n\n- Classical ERM training is inadequate for modeling the variability of dynamical physical processes (parametric PDEs), as demonstrated in section 3.2.\n- An alternative inductive principle is needed. We propose an adaptive conditioning principle: learning from multiple environments to adapt rapidly to a new one (section 3.3).\n\nOn the technical side, while sharing a backbone and using domain-specific weights is common, current meta-learning methods are costly and do not allow learning PDE solvers on large datasets. Thus, our implementation of this principle:\n-  is computationally efficient (fast adaptation to new environments) and scalable w.r.t. the number of training samples via a low-rank and first-order optimization.\n- is adaptable to any NN backbones. Our experiments show it works well with several backbones (MLP, CNN, FNO).\n- can embed physical models learned on different environments, even with incomplete models and unknown parameters, which has never been shown before.\n\n### Motivations of the overall hybrid design\n\nThe proposed model integrates a physics module $g_p$ encoding prior knowledge from a PDE that only partially explains the physical phenomenon. The NN module $g_a$complements $g_p$ by modeling the unexplained physics.\n\nRegarding the combination order ($g_a \\circ g_p$), it is logical that ($g_a$) operates after $g_p$ as it must augment the incomplete knowledge.\n\nAlternative methods exist, such as [1] $g = g_p + g_a$ or [2] a variational formulation. Our approach, $g = g_a \\circ g_p$, is more flexible as it doesn't assume an additive or probabilistic formulation. Given an initial condition $u_0$, $g_p$ outputs an estimate of the temporal derivative $du/dt$, which is then refined by $g_a$.\n\nFor the implementation, the physical component ($g_p$) is encoded as a differentiable numerical solver. This allows it to be combined with any NN components and the entire architecture to be trained end-to-end. Since the physics component has few free parameters, a low-rank formulation is unnecessary. \n\n[1] Yin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, ICLR 2021\n\n[2] Takeishi et al. Physics-Integrated Variational Autoencoders for\nRobust and Interpretable Generative Modeling, NeurIPS 2021.\n\n### What about using it as an adaptor?\nWe added a new baseline “Phys-Adaptor” in the rebuttal (pdf companion Table 3 and 4) where only $g_p$ act as an adaptor, thus $g_a$ is shared across the environments. The lower performance of this baseline illustrates again the importance of adapting $g_a$.\n\n### SVD-PINN as a new baseline.\nPINNs methods are data-free, requiring full knowledge of the physical information, which is not the case here. Our framework is applied on two popular settings:\n\n- Pure data-driven approaches: No prior knowledge of the physics is available, and everything must be learned from data.\n- Hybrid formulation: A PDE partially describing the physical system provides some prior knowledge of the physics (section 3.3.3).\n\nThese settings correspond to common real-world situations explored in the literature.\n\n### Adding an adaptor-based method\n\nOur framework can be used with different backbones as indicated above. [3] has been applied on pre-trained transformers and performs adaptation by adding new “adapter” layers, which increase the number of layers. In our setting, we train a model from scratch to learn to be adapted to new environments efficiently. During adaptation, all baselines keep the same nb. of layers and only tune a small set of parameters, e.g. $c^e$.\n\n[3] Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021\n\n### Experiments with advanced neural operators.\n\nWe added the **Transolver** baseline in our ERM experiments in the rebuttal (Fig 1 and Table 2). The performance is similar to other ERM baselines, verifying that an adaptation framework is necessary regardless of the backbone used.\n\n### Adding training times for the different methods\n\nWe have added the training times and the number of parameters for each method in the PDF rebuttal page (table 1), showing that GEPS shows a huge time gain compared to gradient-based methods (CAVIA baseline) and that our method is less costly in terms of parameters compared to the reference baselines CoDA and LEADS.\n\n### Ensemble baseline with a Transformer-based method.\n\nFor the ensemble method, we used the same model (CNN) as in our GEPS framework for a fair comparison. Besides, the above experiments with the Transolver shows that the backbone itself does not make a significant difference.\n\n### The \"adaptive conditioning\" term is unsuitable.\n\nWe hope to have clarified this issue. The paper main message is that ERM inductive principle alone is not sufficient for modeling multiple complex physical dynamics and that another inductive principle should be used. Hence the adaptation inductive principle developed in the paper.\n\n### Knowledge of the PDE coefficients for ERM experiments\nIn all our experiments, the PDE coefficients are unknown.\n\n### Same knowledge for ERM methods and GEPS?\nThe assumptions are the same for all the methods (baselines and GEPS). The only difference is in the induction mechanism: ERM methods assume that all data come from the same distribution, while GEPS assumes that data come from different distributions and incorporates an adaptation mechanism to adapt the network to each distribution.\n\nWe appreciate your feedback and hope we answered your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SZqwi4nZRK", "reviewer": "Reviewer_1xSb", "summary": "This paper proposes to solve parametric PDEs with the introduction of context parameters. The low-rank design allows for rapid adaptation to unseen conditions, and the experiments show comparable or slightly better performance of the method than baselines.", "questions": "1. Why Neural ODE is utilized to generate the full trajectory? Do the experiment settings require continuous-time inference?", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The utilization of context parameters facilitates efficient adaptation to novel environments while minimizing computational overhead.", "weaknesses": "1. The main architecture of the proposed model is similar to LoRA, despite the fact that the context parameters are learned jointly during the pre-training stage.  The core innovation lies more in the combination of existing techniques rather than introducing a fundamentally new approach. The motivation of the model design is not well established as well.\n2. The experiment results seem not strong, especially for the Gray-Scott equation.\n3. The paper briefly mentions the computational efficiency of the proposed method but lacks a detailed analysis. \n4. The sensitivity of the proposed method to various hyperparameters is not extensively discussed, such as the size of the context vector and the rank in the low-rank adaptation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're thankful for the reviewer's feedback and have addressed the raised concerns below.\n\n### The main architecture of the proposed model is similar to LoRA.\n\nThe paper primarily focuses on the general adaptation framework rather than the LoRA implementation. The key points are:\n\n- Classical ERM training is inadequate for modeling the variability in dynamical physical processes (framed as parametric PDEs), as demonstrated in section 3.2.\n- An alternative inductive principle is necessary; we propose an adaptive conditioning principle: learning from multiple environments to rapidly adapt to a new one (section 3.3). The core issue is developing an effective meta-learning strategy.\n- Our implementation of this principle is computationally efficient (fast adaptation to new environments) and scalable w.r.t. the number of training samples, via low-rank and first-order optimization. \n- We propose a versatile framework compatible with any NN backbone and demonstrate that physical models can be learned across different environments despite incomplete models and unknown parameters, a novel achievement.\n- It improves the state-of-the-art in learning from multiple environments on new proposed datasets. Experiments show our framework works well with MLP, CNN, and FNO backbones.\n\n### The motivation of the model design is not well established as well.\n\nThe proposed model combines a physics module and an agnostic NN module. The former encodes some prior knowledge available under the form of a PDE that only partly explains the underlying physical phenomenon, while the latter comes as a complement to this partial physical model in order to model the physics not explained by the former.\nConcerning the order of the combination ($g_a \\circ g_p$), this order is natural since the NN component $g_a$ comes as a complement to the partial physics encoded in $g_p$.  Alternatives methods exist: [1] propose $g = g_p + g_a$ or [2] proposed a variational formulation. We proposed $g = g_a \\circ g_p$, which is more flexible than existing methods, as it does not assume the physical component needed to be completed additive or a probabilistic formulation. Given an initial condition $u_0$, the physical model $g_p$ takes as input $u_0$ and outputs an estimate of the temporal derivative $du/dt$, which is processed by the data driven term $g_a$. The latter then complements the incomplete estimate computed by the physical component.\n\n[1] Yin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, ICLR 2021\n\n[2] Takeishi et al. Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling, NeurIPS 2021.\n\n\n### The experiment results seem not strong, especially for the Gray-Scott equation\n\nWe have added complementary experiments, including a new state-of-the-art transformer baseline (Transolver) in Figure 1 and Table 2, and a new ablation study, Phys-Adaptor (Tables 3 and 4), which uses a shared NN backbone without adaptation. The Transolver's performance is similar to other backbones, and the ablation highlights the importance of the proposed adaptive framework. Regarding Gray-Scott, we respectfully disagree with your conclusions. Our method outperforms all baselines except APHYNITY in the hybrid setting. Figure 15 qualitatively demonstrates the quality of our predicted trajectories for Gray-Scott.\n\n### The paper briefly mentions the computational efficiency of the proposed method but lacks a detailed analysis.\n\nThanks for the suggestion, we have added in the PDF rebuttal page training time and parameters gains for the different multi-environment frameworks, showing the superiority of the GEPS framework. \n\n### The sensitivity of the proposed method to various hyper-parameters, e.g. context vector size\n\nAgain, this might not be sufficiently indicated in the core paper, but for lack of space, we included the ablation analysis in the appendix. More precisely, in the appendix C.3.1, we studied the impact of the context size on the performance and on the number of parameters required for adaptation. The proposed GEPS reaches higher performance at a lower complexity, in terms of parameters, than the reference baseline CODA. Finally in C.2, we studied how the number of adaptation trajectories impact the adaptation performance.\n\n### Why Neural ODE is utilized to generate the full trajectory? Do the experiment settings require continuous-time inference?\n\nNeural ODE here makes reference to a family of time integration methods. This includes several differentiable PDE solvers. Here we made use of a popular RK4 explicit solver available in the torchdiffeq library. Other alternatives are possible with our framework. For the ERM methods, we used the same time-stepping technique method for GEPS than those used in the ERM methods.\n\nWe hope we clarified all your concerns and will make sure everything will be made clearer in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their comments and suggestions. We are encouraged that they found our method clear and well-written (Reviewer 3TLD, Reviewer m65H). We particularly appreciate that you found we tackle an important challenge (Reviewer m65H) and provide a straightforward yet strong and computationally effective solution (Reviewer 3TLD, Reviewer 1xSb), well evaluated on a variety of datasets against different baselines, showing SOTA or competitive performance (Reviewer 3TLD, Reviewer JETf, Reviewer m65H). We carefully answered all the concerns raised by each reviewer. We address in this section main concerns shared by the reviewers and we introduce additional experiments following reviewers recommendations.

## Additional experimental results - provided in the pdf companion one page: 

- Computational efficiency (All reviewers): Table 1 reports training time and number of parameters for best multi-environment frameworks for Burgers and Gray-Scott equations.

- Error bars (Reviewer JETf): we report error bars for in-range time horizon (out-range horizon has been removed due to lack of space) for both in-domain (Table 3) and out-domain (Table 4) experiments, showing that our results are consistent and significant, training all different frameworks on 3 different seeds.

- Physical adaptor baseline (Reviewer 3TLD, Reviewer m65H): we propose a new baseline “Phys-Adaptor” in table 3 and 4 following reviewer m65H recommendation where only the physical component is adapted and the neural network is shared across all environments. The lower performance of this setting demonstrates the importance of our low-rank adaptation mechanism (Reviewer 3TLD).

- **Transolver** baseline for ERM methods (Reviewer m65H): we have added an advanced Transformer model to our ERM baselines following Reviewer m65H  recommendation for both IVPs in Fig. 1 (Burgers has been removed for lack of space, but similar results have been observed) and when considering a historical window of past states as input (Tab 2), confirming our observations that whatever  the NN backbone used, ERM methods are not effective for multi-environments datasets.

## Main concerns

### Novelty of the adaptation mechanism (Reviewer m65H, Reviewer 1xSb):
We want to highlight that the focus of the paper is on introducing a general adaptation framework suitable for modeling complex PDEs, and not on the implementation of a specific NN model as some of the remarks suggest.  The main messages are:

- Classical ERM training is inadequate for modeling the variability in dynamical physical processes (framed as parametric PDEs), as shown in section 3.2.
- As an alternative, we propose an adaptive conditioning principle: learning from multiple environments to rapidly adapt to a new one (section 3.3). The key focus is developing an effective meta-learning strategy.
- Our implementation of this principle is computationally efficient (fast adaptation to new environments) and scalable w.r.t. number of training samples via a low-rank and first-order optimization formulation. We demonstrate that this model improves the state-of-the-art for learning from multiple environments.
- This framework is adaptable to different NN backbones. Experiments show that it works well with MLP, CNN, and FNO. We also demonstrate that physical models can be embedded and learned across different environments, even when incomplete models and with unknown parameters, which has not been shown before.
- We have validated the method on three new datasets, with multiple environments generated using a large diversity of parameters such as  pde coefficients, domain definition or forcing terms.

### Model design choice (Reviewer JETf, Reviewer m65H): 
- For the hybrid modeling setting, the proposed model integrates a physics module and an agnostic NN module. The physics module encodes prior knowledge from a PDE that only partially explains the underlying physical phenomenon, while the NN module complements this by modeling the unexplained physics.

- Regarding the combination of the physical and data-driven modules ($g_a \circ g_p$), this order is logical because the NN component ($g_a$) complements the partial physics encoded in ($g_p$). Alternative methods exist, such as [1] $g = g_p + g_a$ or [2] a variational formulation. We propose $g = g_a \circ g_p$, which is more flexible as it doesn't assume an additive or probabilistic completion of the physical component.

- For time integration, we considered NeuralODE, referencing a family of time-stepping methods. In our implementation, we used a differentiable RK4 numerical solver, though other solvers could also be used. For a fair comparison with ERM methods, we used the same time stepping method proposed in the corresponding reference papers for GEPS.

We acknowledge that Figure 4 in the paper might imply our framework works only with NeuralODE. We will clarify this in the final version.

[1] Yin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, ICLR 2021

[2] Takeishi et al. Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling, NeurIPS 2021.

### Computational efficiency of our method (All reviewers)

- We have added the training times and number of parameters for all the methods in table 1 for the Burgers and Gray-Scott equations. We observe that GEPS is more efficient in terms of training time and in terms of parameter gains. 
- In the main paper (c.f. Fig. 12 and Table 8 in the appendix), we showed that our method performs and scales better in terms of accuracy and total training parameters when varying the context vector size compared to reference CoDA. 
- Also, in Fig 13. in appendix, we observe that our method adapts faster (less steps) to new environments than CoDA.

We hope this clarifies the main concerns pointed out by reviewers and will make sure that the different points discussed and new experiments will be added in the final version.

Meta-review: The paper proposes a novel method, GEPS, to improve the generalization of neural PDE solvers to unseen dynamics. GEPS employs a low-rank meta-learning strategy to enable efficient adaptation to new environments by adjusting only a small set of context parameters. The method incorporates physics knowledge through a hybrid framework, combining a physics module with a data-driven module. Extensive experiments demonstrate the effectiveness of GEPS in various spatio-temporal forecasting tasks.

Strengths:

The paper is easy to follow, with clear explanations and concise language.
The low-rank meta-learning strategy for adaptive conditioning is a novel contribution to the field of neural PDE solvers.
The authors provide extensive experiments to validate the effectiveness of GEPS, demonstrating its superiority over existing methods in many cases. 
The incorporation of physics knowledge through a hybrid framework offers a flexible approach to modeling complex PDEs.

Weaknesses:

While the low-rank meta-learning strategy is novel, the overall approach of using a hybrid framework and domain-specific weights is not entirely new.
The paper could provide a more detailed explanation of the motivation behind the specific design choices, such as the use of a prior module.
Some important baselines, such as adaptor-based methods and advanced neural operators, are missing from the experiments.

Recommendation:

The paper presents a promising approach for improving the generalization of neural PDE solvers. While the technical novelty is low, the paper's strengths in terms of clarity, empirical results, and hybrid framework outweigh the weaknesses. All the reviewers vote for acceptance. I, therefore, recommend accepting the paper and encourage the authors to use the feedback provided to improve the paper.
Decision: Accept (poster)