MetaPhysiCa: Causality-aware Robustness to OOD Initial Conditions in Physics-informed Machine Learning
Authors: S Chandra Mouli
Bruno Ribeiro
Venue: Submitted to ICLR 2023 (2023)
Keywords: physics-informed machine learning
out-of-distribution
robustness
causality

Abstract: A fundamental challenge in physics-informed machine learning (PIML) is the design of robust PIML methods for out-of-distribution (OOD) forecasting tasks, where the tasks require learning-to-learn from observations of the same (ODE) dynamical system with different unknown parameters, and demand accurate forecasts even under initial conditions outside the training support. In this work we propose a solution for such tasks, which we define as a meta-learning procedure for causal structural discovery (including invariant risk minimization). Using three different OOD tasks, we empirically observe that the proposed approach significantly outperforms existing state-of-the-art PIML and deep learning methods.

Under review as a conference paper at ICLR 2023
METAPHYSICA: CAUSALITY-AWARE ROBBSTNESS TO OOD INITIAL CONDITIONS IN PHYSICS-INFORMED MACHINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A fundamental challenge in physics-informed machine learning (PIML) is the design of robust PIML methods for out-of-distributie (OOD) forecasting tasks, where the tasks require souminn-to-leama from observations of the same (ODE) dy- namical system with different unknown parameters, and demand accurate forecasts even under initial conditions outside the training support. In this work we propose solution for such tasks, which we define meta-learning procedure for causal structural discovery (including invariant risk minimization). Using three differ- ent OOD tasks, we empirically observe that the proposed approach significantly outperforms existing state-of-the-ard PIML and deep learning methods.
INTRODUCTION
Physics-informed machine learning (PIML) (e.g., (Willard et al., 2020; Xingjian et al., 2015; Lusch al., 2018; Yeo & Melnyk, 2019; Raissi et al., 2018; Kochkov et al., 2021)) seeks combine the strengths of physics and machine learning models and has positively impacted fields as diverse as biological sciences (Yazdani al., 2020), climate science (Faghmous & Kumar, 2014), turbulence modeling (Ling et al., 2016; Wang et al., 2020a), among others. PIML achieves substantial success in tasks where the test data comes from the same distribution as the training data (in-distribution tasks).
Unlike the PIML work described above, this paper considers an out-of disttributittn (OOD) change in the initial system state of the dynamical system, possibly with different train and test distribution supports (illustrated in Figure ((a.b)). In this setting, we observe that existing sate-offthhe-ar PIMI. models perform significantly worse than their performance in-distribution, even PIML methods designed with OOD robustness in mind (Wang et al., 202 Kirchmeyer et al., 2022). This is because the standard ML part of PIML, which tends to learn spurious associations, will perform poorly in our OOD setting. We then propose a promising solution: Combine meta learning with causal structure discovery to learn an ODE model that is robust to OOD initial conditions. In our OOD tasks, OOD robustness means that the robustness is tied to interventions over the initial conditions of the system, not on arbitrary interventions as the system evolves from the initial state. This is an important distinction. There can be multiple ODE models that will be equally OOD robust, and robust ODEs may not correctly predict system trajectories under arbitrary system interventions besides the initial state (Rubenstein al. (2016) discusses the effect of arbitrary interventions in physics models).
Contributions This work proposes hybrid trunsductive-induer
modeling approach learning for more robust ODEs using meta learning and causal structure discovery (c.g., via L1 regulariza- tion (Zheng et al., 2018), which can be combined with invariant risk minimization (Arjovsky et al., 2019; Krueger et al., 2021)). More precisely, our contributions are:
1. We show that state-of-the-at PIML and deep learning methods fail in test examples with OOD initial conditions. Prior work (Wang al., 202 showed that deep learning-only methods fail in OOD tasks, and argued physics models and PIML methods would succeed, including a proposed OOD solution (Wang et al., 2021b). Here show that PIML methods also fail (or perform poorly) OOD, including the solution in Wang et al. (2021b).
2. We proposed a hybrid iotssduuttve-innuccee learning framework for ODEs via meta learning: As in transductive methods, we will consider each training and test examples separate tasks, but

---

Under review as a conference paper at ICLR 2023
like inductive methods, the tasks are dependent and knowledge be transferred between the learned ODEs. By meta learning we mean the definition in (Thrun & Pratt, 1998, Chapter 1.2), where given: (a) a family of M tasks (a task is single experiment in our setting), i ...... M; (b) training experience for each task i E {1,..., M). which for us are the time series observations of experiment X ....,..;,
X ty. and; (c) family of performance measures (e.g., for each task) described by the risk function R(); our algorithm will meta learn such that performance at each task improves with experience (more observations) and with the number of tasks (number of experiments). For an algorithm to fit this definition, there must a transfer of knowledge between multiple tasks that has positive impact expected task performance across all tasks.
Learning ODEs as structural causal discovery. In order learn an ODE that is robust OOD changes in initial conditions (with possibly non-overlapping training and test distribution supports), we define a family of structural causal models and perform structural causal search in order to find the correct model for our task (which is assumed to be in the family). We test common structural causal discovery approaches for linear models: & Pr-reularization with and without an invariant risk minimization-type objective, which observe achieve similar empirical results.
The proposed method is then empirically validated using three commonly-uses simulated physics tasks (with measurement noise): Damped pendulum systems (Yin et al., 2021), predator-prey sys- tems (Wang al., 2021a), and epidemic modeling (Wang et al., 2021a), all under both constant ODE parameters and varying ODE parameters per experiment. ODE parameters between train and test experiments have the same distribution but non-overlapping (OOD) initial condition distributions.
2 DYNAMICAL SYSTEM FORECASTING AS A META LEARNING TASK
In this section formally describe the task of forecasting a dynamical system with a focus on the out-of-distribution initial condition scenario.
Definition (Dynamical system forecasting task) In what follows describe task:
1. Training data (depicted in Figure I())): training, we are given a set of M experiments, which we will denote as M tasks. Task i (1...., M) has associated (hidden) environment e
Different tasks can have the same environment. Let T() == x(9) x()) denote the noisy
7600
observations of our dynamical system, with x xt (3)
+ Et where
dx
dt (a W (00,
(I)
(oos... regularly-ppaced discrete time steps, x. Rdd is the (hidden) state of the system at time t during experiment (task) i, et are independent zero-mean Gaussian noises, vo is an unknown deterministic function with hidden ground truth parameters W ~ P(W*) and funtritoosticcunntoon
E", where the global lask- musk-independent parameters W* and g" are also hidden. Regularly spaced intervals are not strictly necessary for our method, but it makes its implementation simpler. Initial conditions: The distribution of initial conditions x(i) P(Xio E = =!!)) of task i may depend on its environment. The unknown parameters e* remain constant across environments.
Test data ((depicted in Figure 1(b)): At test, we are given an observed initial sequence (M++1) = x (M+1)
(M+1) where r is generally small, of the dynamical system
to
day dal (M+1)
(i (M+1), W(M+1)+ ;"")
dt
(M+1) x(M+1) P(X., E E= ((+1)) and (unknown) system parameters with initial condition x
WM++1)+ P(W*) and hidden global parameters E" the same as training. Our task is
to predict ((++)) ...., x...,x(M FTMM from the initial observations (M+1), using the inductive knowledge obtained from the training data.
x(M+1)
2

---

Under review as a conference paper at ICLR 2023
3. Out-of-distribution initial conditions: Initial conditions in training {P(Xto | ) can be different from initial conditions in test P(X., E = = with possibly non-overlapping support due to the presence of an unseen environment in training.
In training, we are given trajectories that may have (a) different initial conditions, and (b) different unknown ODE system parameters. We observe a test trajectory (indexed by M + 1) from time t= to....,.. by and wish to forecast its future after time fy. The test trajectory can have an OOD initial condition but in-distribution ODE parameter W(M++1)+ with unknown value.
Illustrative example. Figure 2a shows an example of an out-of-distribution task for forecasting the motion of a pendulum with friction. (1.) The state X, [0r, wi] E IR² describes the angle made by the pendulum with the vertical and the corresponding angular velocity at time t. The true (unknown) function v describing this dynamical system is given by (10,, W = [W -a sin( ) p" with W = (a",, pp)) denoting the parameters relating to the pendulum period and the damping coefficient. (2.) In training, we observe M (noisy) trajectories of motion over discrete time steps t 0, ... from experiments (tasks) where pendulum dropped with no angular velocity. (3.) In training. each experiment performed by dropping different pendulums (i.e., ( P(W*))
from angles 0 < 82, x /2. (4.) In test, the experiment is repeated with a different distribution over the initial dropping angles, π 0.1 < 0t < т (nearly vertical angles) with small angular velocities.
The test trajectory is observed over smaller time window / = 0, 0.1, 3.3 and the forecasting task to predict the future states of the pendulum till time t = 10.
(a) Observations from unknown dynamical system:
(b) Predict future states same dynamical system with OOD initial states.
Task i
dx ())
w
OOD initial
dt
state
Test Task
())
Predict
x
(radiann))
Task i = 2
fradenss
(3)
4o
Timee
Time
(c)
Standard Neural Networks
(d) Transductive PIML
(e)
Inductive PIML
Poor OOD approximation with standard activations (ReLU, Tanh)
Does not transfer knowledge
Neural network term & physics parameter prediction are not robust OOD
from training tasks to test task
Pendulum motion: NeuralODE
predictions of vs ground truth
(radiand)
Training
Suga
Test
Teee
model
Physics model :ww Predict physics parameters for each task
Neural Network
Time
Φ(
Predict
+
Learn corrections to
test
PIML
physics model
Model
Figure 1: Dynamical system OOD problem definition and traditional approaches to address it. (a) Training data consists of multiple observations from the same dynamical system with different parameters (++ Each training curve can be seen different task i where the goal test, we are given observations till f, (red solid) and the goal to predict the future observations till tr (gray dashed). (c) Shows OOD failure of a standard neural network (NeuralODE. (Chen et al., ()18)) for dynamical system forecasting. When trained to predict the motion of damped pendulum, the model predicts accurately in the training domain (green shaded), but predicts a linear function outside the training domain. (d) Transductive PIML methods (e.g.. (Raissi al., 2017a: Brunton al.. 2016)) not able to transfer knowledge from training tasks to a test lask with different W Thus, these models can fit only using test observations ignoring the training data. (e) Inductive PIML methods (e.g., (Yin et al., 2021; Mehta et al., 202 use known (possibly incomplete) physics model d( ;w) and inductively predict its parameters for each task, typically using neural network. However, predicting these physics parameters test this way is not robust. Furthermore, they use neural network term to correct the incomplete physics model and face same robustness issue discussed (c).
predict X from x(") for all t. (b) At
4,
3

---

Under review as a conference paper at ICLR 2023
nearly vertical
acute
())
(i))
(a) Pendulum task
(b) In-distribution predictions of
(c) OOD predictions of
Test NRMSE |
Constant W ID
Varying W ID
Methods
OOD
OOD
Standard Deep Learning
NeuralODE (Chen et al., 2018)
0.016 (0.008)
0.739 (0.056)
0.083 (0.033)
0.591 (0.119)
DyAd (Wang et al., 2021b)
0.012 (0.000)
0.695 (0.049)
0.078 (0.051)
0.834 (0.263)
CoDA (Kirchmeyer et al., 2022)
0.017 (0.011)
0.669 (0.365)
0.052 (0.032)
0.764 (0.201)
Physics-informed Machine Learning
APHYNITY (Yin et al., 2021)
0.047 (0.022) NaN*
0.803 (0.168) NaN
0.097 (0.020) NaN*
0.970 (0.384)
SINDY (Brunton et al., 2016)
NaN'
EQL (Martius & Lampert, 2016)
NaN*
NaN*
NaN
NaN 0.070 (0.011)
MetaPhysiCa (ours)
0.028 (0.005)
0.078 (0.090)
0.049 (0.002)
(d) Normalized RMSE of test predictions from different methods in two cases: with W(1+) constant or varying across environments NaN indicates that the model returned errors during test-time predictions, for example, because the learnt ODE was too stiff (numerically unstable) to solve.
Figure 2: (a) Predict pendulum motion from noisy observations: (i) in di-diitribution, when dropped from acute angles and (ii) cuttof-diisributionn when dropped from nearly vertical angles. (b, e) shows example ground truth curves (blue stars) and out-of-distribution along with predictions from different models. While most tested methods perform well in-distribution, only MetaPhysiCa (orange) closely follows the true curve OOD and all other methods are terribly non-robust. (d) Standard deep learning methods and physics-informed machine learning methods fail to forecast accurately distof-diitribution. On the other hand, MetaPhysiCa outputs up to 8.5x more robust OOD predictions.
3
RELATED WORK & CHALLENGES WITH EXISTING APPROACHES
Next we describe different classes of existing approaches that are commonly used for the dynamical system forecasting (Definition 1) and their inherent challenges dittiibribt
3.1 STANDARD NEURAL NETWORKS METHODS
Deep learning's ability to model complex phenomena has allowed it to make great strides in number of physics applications (Lusch et al., 2018; Yeo & Melnyk, 2019; Kochkov et al., 2021; Dang et al., 2022; Brandstetter al., 2022b). However, standard deep learning methods are known learn spurious correlations and tend to fail when the test distribution of the inputs are different from that observed in training (Wang al., 2021a; Geirhos et al.. 2020). Figure depicts the -iitt--distribttion failure of several deep learning methods from NeuralODE (Chen et al.. 2018) to more complex meta learning approaches (Wang et al., 2021b; Kirchmeyer et al., 2022) in our running damped pendulum example (more details of the experiment is in Section 5).
In standard deep learning tasks, Xu et al. (2021) show that an MLP's failure to extrapolate to out-of- distribution can be traced to an absence of algorithmic alignment, which is an appropriate combination of basis and activation functions within the architecture for the task. For example, the outputs of an MLP with ReLU activations will be linear far from the training domain even when trained to predict a sine/quadratic function. For dynamical system forecasting, our Figure 1(c) depicts the results of similar experiment for standard sequence model (NeuralODE): the model can approximate the target sine function in the training domain (green region) but predicts linear function far outside the training domain. This means that PIML also needs algorithmic alignment (i.e., include appropriate basis functions) in order to make accurate forecasts in OOD tasks.
3.2
PIYSSICS-IFFOREDD MACHINE LEARNING (PIML) METHODS
To alleviate the challenges described above for standard neural networks, several thysics-informen machine learning (PIML) methods have been proposed (e.g.. (Willard et al., 2020; Wang al., 2020a; Faghmous & Kumar, 2014; Daw et al., 2017)) that utilize physics-basee based domain knowledge about the dynamical system in consideration for better predictions. The type of physics- based knowledge vary
4

---

Under review as a conference paper at ICLR 2023
across methods, for example, (a) a dictionary of basis functions (e.g., sin, cos, de (Schmidt & Lipson, 2009; Brunton et al., 2016; Martius & Lampert, 2016; Raissi, 2018: Cranmer et al., 2020a) related to the task. (b) a completely specified physies model (Raissi et al., 2017a; Raissi, 2018: Jiang et al., 2019) or with missing terms (Yin et al., 2021). and (c) different domain-specific physical constraints such energy conservation (Greydanus et al., 2019; Cranmer al., 2020b), symmetries (Wang et al., 2020b; Finzi et al., 2021; Brandstetter et al., 2022a). While these PIML methods improve upon standard neural networks, Figure shows that they are generally not designed for OOD forecasting tasks. To precisely study the reasons for this failure, we categorize these methods into inductive and transductive methods based on requirements over the dynamical system parameters W
Transductive PIML methods. Transductive inference focuses on predicting missing parts from the training data. In PIML., transductive inference methods treat each training and test examples as unrelated tasks, hence OOD generalization tends to be less of a challenge in transductive methods. For instance, SINDY (Brunton et al.. 2016), EQL. (Martius & Lampert, 2016), and related methods (Raissi, 2018; Chen, 2021), learn the ODE equation based on dictionary of basis functions for specific parameter ( These transductive methods, however, do not transfer knowledge learnt in training to predicting test examples with a different in-distribution W This forces these methods to forecast simply based on the initial observations of the test task alone, often leading to poor performance. Figure ((d) illustrates this case where a transductive method (unsuccessfully) tries to learn the unknown parameter (33)* of the test task from few initial test observations. Another class of transductive methods (Raissi et al., 2017a;b: Yu al., 2022) assume all the physies parameters W* of all experiments to remain constant across all training and test tasks, regularizing neural networks to respect a given physics model. They have been shown be challenging to train for harder differential equations ( (Krishnapriyan et al., 2021) or return trivial solutions (I (Leiteritz. & Pflüeer, 2021). Recently, Causal PINNs (Wang et al., 2022) mitigate some of these training challenges by ensuring that, for any time f, predictions at time less than are accurately resolved before predictions at time t. Not only will these methods perform poorly in-distribution if different experiments have different physics parameters, they also do not allow for causal interventions variables in the dynamical system.
Inductive PIML. Taking the opposite approach, inductive inference focuses on learning rules from the training data that can be applied to unseen test examples. Inductive methods dominate PIML approaches but are fragile OOD, since the learned rules are learned within the scope of the training data and are not guarantee work outside the training data scope. For example, APHYNITY (Yin et al., 2021) and NDS (Mehta et al., 2021) are such inductive methods that augment a neural network to a known incomplete physics model where the parameters of the physics model are predicted inductively using a recurrent network. As illustrated in Figure I(e), these methods are able to learn from training tasks with different true parameters W However, in our experiments, APHYNITY often returns incorrect physics parameters OOD (see Figure 2c). Further, the augmented neural network suffers from the same issues discussed in Section 3.1 leading to poor OOD performance as seen in Figure 2.
With these key reasons identified for the fragility of existing methods to OOD initial conditions, next we propose an approach (MetaPhysiCa) that is more robust to these challenges and outputs more robust predictions out-of--istribuiie while also giving accurate predictions in mn-distribttin..
PROPOSED APPROACH: METAPHYSICA
f fi
In what follows we describe MetaPhysiCa, our pro- posed approach. We start with the description of family of causal models, then explain how meta learning allows us to perform a hybrid transductive- inductive approach for improved OOD accuracy.
E
W
W
Not causally related future states
4.1 STRUCTURAL CAUSAL MODEL
We describe the dynamical system using determin- istic structural causal model (Peters et al., 2022) with measurement noise over the observed states and ex- plicitly define the assumptions over the unknown function di in Definition 1.
dbs dlt
Figure 3: Deterministic SCM for a dynam- ical system with measurement noise. The dynamics is defined via an unknown linear combination of basis functions.
5

---

Under review as a conference paper at ICLR 2023
The causal diagram is depicted in Figure 3 in the plated notation iterating over time t t= tu. ........ trr for each task T(1) As before, the state of the dynamical system x E R for task i. We note that our SCM may not necessarily be the true SCM, but rather SCM that is indistinguishable from the true one w.r.t. interventions limited to changes in the environment variable E that affects the initial conditions
x(O We define the causal process at each time step t for i-th task as follows. Let fx( Rd R, I k S m, be linearly independent basis functions each with separate set of parameters Ex² acting on an input state zi Examples of such basis functions include trigonometric functions like fi (zx (1) ; ii f1,2), polynomial functions like fa
x E2) 2f, and so The corresponding outputs from these basis are shown as (6) == fa (0), Ex) in Figure 3. The derivative d=(3/ de for particular dimension { d} Fke
(1) is only affected by few (unknown) basis function outputs =k,t (green arrows in Figure 3) and is linear combination of these selected basis functions with coefficients ( However, these selected basis functions and their corresponding parameters & are assumed be invariant across all the tasks, i..., dt, j E (l...., , d), defined using the same basis functions for all i = I,..., ......,... M. Finally, the derivatives dictate the next state of the dynamical system. We observe the dynamical system with (8) independent additive measurement noise X. = x.t + + where e,") o
We assume that we are given the collection of m possible basis functions
1....., m,
m 2 2, with unknown & and no prior knowledge of which {fx causally infturence dæl /at. The need for basis functions stems from extensive experimentation and our analysis in Section 3.1, where we show that appropriate basis functions must be incorporated within the architecture in order to extrapolate to OOD scenarios (see Figure I(c)).
4.2 META LEARNING & MODEL. ARCHITECTURE
Given the training data {(ar ) i=] generated from the unknown SCM described above, our goal is three-fold: (a) discover the true underlying causal structure, i.e., which of the edges 21,t - dat./dt exist for j 1, ... d, (b) learn the global parameters & that parameterize the relevant basis functions, and (c) learn the task- speciifc parameters W()+ that act as coefficients in linear combination of the selected basis functions. In the following, we propose a meta-learning framework that introduces structure (gate) parameters & that are shared across tasks and task-specific coefficients w that vary across the tasks
(0)
M
dx{ (W @ (
dt
where O is the Hadamard product and
F [x{ (0: = (
fm (X
Em)
T
is the vector of outputs from the basis func-
tions with parameters E
T {0, 1) dxx are the learnable parameters governing the global causal structure across all tasks such that 2 = 1 iff edge 2,,t - dais/dt exists in Figure 3,
(i0) pdxm are task-specific parameters that act as coefficients in linear combination of the selected basis functions.
Next we describe a procedure to obtain the structure parameters 4. Finding whether an edge exists or not in the causal graph is known as the causal structure discovery problem (e.g.. Heinze-Deml et al. (2018)). We use a score-based causal discovery approach (e.g.. Huang et al. (2018)) where we assign a score to each possible causal graph. We wish find the minimal causal structure, i.c. with the least number of edges, that also fits the training data. This balances the complexity of the causal structure with training likelihood, and avoids overfitting the training data. A sparse structure for t implies fewer terms in the RHS of the learnt equation for the derivatives in Equation (2). Several causal discovery approaches have been proposed that learn such minimal causal structure via continuous optimization (Zheng et al., 2018; Ng et al., 2022). We use the log-likelihood of the training data with L reggularization term to induce sparsity that is known to perform well for general causal structure discovery tasks (Zheng et al., 2018). Note that since the direction of all the edges known (i.e., Zk.t - daat)/a)), we do not need the acyclicity constraints and the causal graph is uniquely identified by its Markov equivalence class (Pearl, 2009, Chapter 2).

---

Under review as a conference paper at ICLR 2023
The prediction error is given by R (W(6, 2. T E
x where X
X + S. (W o ( are the predictions obtained using an ODE solver to integrate Equation (2). In practice however, we found the squared loss directly between the predicted and estimated ground truth derivatives, i... R
L, () TO 2i=ta ||ax; /dt "x' /ael leads to stable learning procedure with better accuracy in-distribution and OOD. As discussed before, we use an e-rreuuaariztion term || to learn a causal structure with the fewest possible edges 2k,t - /t.. = 1, l..... d, while minimizing the prediction error in training. We also use l1 er-regularization on the task-specific parameters, [ to learn simpler model within each task i, if possible, than the one learnt globally for all tasks via 4.
Our structure discovery task comes with additional challenge as the training tasks could have been obtained under different (hidden) environments (as defined in Definition 1). While there are score-based (discrete optimization) approaches (Ghassami al., 2018; Perry al., 2022) for such non-lID data, aforementioned approaches based on continuous optimization (c.g., (Zheng et al., 2018)) are not guaranteed learn the correct structure. For example, they may output structure that is optimal for one environment consisting of large number of training tasks but suboptimal for other environments. Our goal then is to learn a structure that minimizes the prediction error across all environments simultaneously, similar to learning robust representations via invariant risk minimization- type methods (Arjovsky et al., 2019; Krueger et al., 2021). Since the environment e of a particular task is hidden to our approach, we use a modified V-REx regularization (Krueger et al., 2021) that minimizes the variance of prediction errors across tasks instead of environments, focusing on robustness to the worst-case scenario (that all tasks have unique environments).
ready to describe our final optimization objective. Similar to standard meta-learning objectives (Finn et al., 2017; Franceschi et al., 2018; Hospedales et al., 2021), we propose a bi-level objective that optimizes the structure parameters & and the global parameters in the outer-level, and the task-specific parameters (
Now
in the inner-level as follows
Φ arg min 3.
Σ
R (W 2) + + AREX Variance 2 ())
M
s.t. W
arg min R (w (1) 2) + | vi= I,..,...,AN
(3)
W
where Xq. Aw and AREX are hyperparameters. While the exact bi-level optimization in Equation (3) is challenging solve due the lack of closed-form solution for the inner optimization, it can be approximated by alternate SGD steps for (Φ,E) and ( in outer and inner loops re- spectively (Borkar, 1997: Chen et al., 2021). In our experiments, jointly optimizing & and W = 1,.... M, instead resulted comparable performance with considerable computational benefits over alternating SGD. The discrete structure parameters & can be approximated using (stochastic) Humbel-Softmas variables (Jang et al., 2017: Ng et al., 2022) or using deterministic binarization techniques (Courbariaux et al., 2015; 2016). We use the latter and reparameterize
$jA : ((a(#).) j.a) > 0.5) where &' Rdxm (-) is the sigmoid function, and the gradients estimated via a straighs-trrouh--ssimaat
Hyperparameter selection: We choose the hyperparameters Xo Xw, ARES that result in sparsest model (i.e., with the least [1o11o) while achieving validation loss within 5% of the best validation loss in held-out in- in-distribution validation data. The use of in-distribution data for validation is key requirement since in OOD tasks one does not have access to samples from the test distribution. Additional implementation details are provided in Appendix B.
4.3 TRANSDUCTIVE TEST-TIME ADAPTATION WITH INDUCTIVE REGULARIZATION
((M+1) "..,,,,((+.+) with the unknown ground-truth pa- rameters (M++))+ ~ P(W*) as defined in Definition 1, we adapt the learnt model's task-specific parameters (M++1) by optimizing the following while keeping d غ fixed
Finally, given test task (M+1) (X(M++1) ,,..,,.,
to
tx
(M+++) = arg min
(M+1) x- (M++)) 113 + Aw W (M+1
(4)
W(M+D + 1=to
7

---

Under review as a conference paper at ICLR 2023
Test Normalized RMSE (NRMSE) 4
Constant W(
Varying W
Methods
ID
OOD
ID
OOD
Standard Deep Learning
NeuralODE (Chen et al., 2018)
0.012 (0.002)
1.098 (0.
0.193 (0.024)
1.056 (0.141)
DyAd (Wang et al.. 2021b)
0.016 (0.003)
1.213 (0.216)
0.244 (0.025)
1.088 (0.373)
CoDA (Kirchmeyer et al., 2022)
NaN*
NaN*
NaN*
NaN
Physics-informed Machine Learning
APHYNITY (Yin et al., 2021)
0.047 (0.019) NaN*
0.301 (0.139) NaN*
0,421 (0.332) NaN*
3.937 (1.686)
SINDY (Brunton al., 2016)
NaN
EQL. (Martius & Lampert, 2016)
NaN
NaN
NaN
NaN'
MetaPhysiCa (Ours)
0.008 (0.001)
0.008 (0.000)
0.049 (0.008)
0.129 (0.030)
(a) Test NRMSE + for different methods. NaN indicates that the model returned errors during test.
Petai
NeuralDDE
Aphynity
SINDy
DyAd
EQL
True
10.0
Prodator
Proy
Predstor
7.
5
5
10
0.0
25
5.0
7.3
10.0
3.0
0.0
2.5
3.5
75
15.0
(b) In- distribbbbtton predictions
(c) OOD predictions
Figure 4: (Predator-pree results) (a) MetaPhysiCa outputs 30x and 8x more robust OOD predictions in constant ( and varying W datasets respectively. (b, shows example ground truth curves (blue stars) in- and -fitt--diitiibuttion along with corresponding predictions. While most tested methods perform well in-distribution, only MetaPhysiCa (orange) closely follows the true curve OOD.
where X (M+1) = x (M++1) + +S., (W((W))) @ ()) (M+1), ; E) are the predictions obtained using
the optimal values $ E. and dw is the hyperparameter chosen during training. Note the following two key aspects of the test-time adaptation in Equation (4): (a) Only the task-specific parameters (M++1) are adapted whereas the meta-model & learnt during training kept fixed, and (b) only the observations from time to...., by of the given test trajectory is used to adapt the parameters (M+1), The final predictions (X, (M)++ (M(())))) from the model are obtained with the test-time )2r
adapted parameters (M++) and the fixed parameters with no adaptation J
5 EMPIRICAL EVALUATION
We evaluate MetaPhysiCa in synthetic forecasting tasks based on different dynamical systems (ODEs) from the literature (Yin et al., 2021; Wang et al., 2021a) adapted to our OOD scenario, namely, (i) Damped pendulum system, (fi) Predator-prey system and (iii) Epidemic model. We compare against the following approaches: (a) NeuralODE (Chen et al., 2018), deep learning method for learning ODEs. (b) DyAd (Wang et al.. 2021b) (modified for ODEs), a meta-learning framework that adapts the forecaster to different training tasks with weakly-supervin (c) CoDA (Kirchmeyer et al., 2022), that learns to modify its parameters to each environment with a low-rank adaptation, (d) APHYNITY (Yin et al., 2021). PIML method that augments a known incomplete physics model with a neural network, (c) SINDy (Brunton et al., 2016), a transductive PIML method that uses sparse regression to learn linear coefficients over a given set of basis functions, (f) EQL (Martius& & Lampert, 2016). a transductive PIML method that uses sin, cos and other activation functions within a neural network and learns sparse model. Additional details about the models is presented in Appendix B.
encoder,
Dataset generation. As per Definition for each dynamical system, we simulate the respective ODE to generate M = 1100 training tasks each observed over regularly-spaced discrete time steps (to....,ty ......tr) fr where v/,4 0.11. For each training task (+), i = 1,..., M,, we sample an initial P(X,, E e) where
e is the training environment. We consider two scenarios for the dynamical system parameters: (a) Constant ( where W(() is constant for all tasks i, and (b) Varying (i)),, where we sample a different ( ~ P(W") for each task i. Note however that none of the models have oracle knowledge of which of the two scenarios the data is
condition x(()
In our experiments, we let (+) T constant for all tasks for simplicity of implementation hut the proposed method is not restricted this case.
8

---

Under review as a conference paper at ICLR 2023
observed from. At OOD test, we generate M' = 200 test tasks by simulating the respective dynamical system over timesteps (to..... ). where again ll, tr = 0.1l. For each test task j = .,........ AM' we e') where E e' is the test environment and can induce completely different support for the initial conditions X than in training. The distribution of the dynamical system parameters W is kept the same across training and test.
sample OOD initial conditions X!6o P(X,,
We consider three dynamical systems in our experiments, with 3 to 6 RHS terms in their respective differential equations: a damped pendulum system (Yin et al., 2021), predator-prey system (Wang et al., 2021a), and an epidemic (SIR) model (Wang et al., 2021a), with following OOD shifts in their initial conditions respectively: acute initial angles in training to nearly vertical initial angles in OOD test, initial prey population x less in OOD test than in training, and initial population susceptible to a disease 10x more in OOD test than in training. We generate the damped pendulum dataset with 1% zero-mean Gaussian noise and the rest with noise to show that OOD failure of baselines is unrelated to noise: existing methods fail OOD even with clean observations. For methods that require ground truth derivatives during training, we estimate them from noisy trajectories using Total Variation Regularization (TVR) (Rudin et al., 1992; Chartrand, 2011) 1) as done by Brunton al. (2016). Detailed description of the datasets is presented in Appendix A and experiments with increasing amounts of noise is presented in Appendix C.4.
Results. We repeat our experiments 5 times with different random seeds and report in-distribution (ID) and auloof-distribution (OOD) normalized root mean squared errors (NRMSE), i.e., RMSE normalized with standard deviation of the ground truth observations. Figures 2, 4 and 5 show the errors and example predictions from all models for the three datasets respectively.
The first two columns of Tables 2d, 4a, 5a show results when ( is constant across tasks i. NeuralODE, DyAd, CoDA and APHYNITY use neural network components and are able to learn the in-distribution task well with low errors. However, the corresponding errors OOD are high as they are unable to adapt to OOD initial conditions. Example OOD predictions (Figures 2c, 4c and 7b) from these methods show that they have not learnt the true dynamics of the system. For example, for epidemic modeling (Figure 7b), most models predict trajectories very similar training trajectories even though the number of susceptible individuals is 10x higher in OOD test. SINDY and EQL cannot use the training data and are fit on the test observations alone (see Figure (d)). Thus, they are unable to identify an accurate analytical equation from these few observations of the test task, resulting in prediction issues due to stiff ODEs. MetaPhysiCa consistently performs the best OOD across all datasets achieving 8.5x to 35x lower NRMSE OOD errors respectively in the 3 datasets than the best baseline. The last two columns of Tables 2d, 4a, Sa show results for the more challenging scenario when W ~ P(W*) ) is varying across tasks. The results follow the same trend and MetaPhysiCa performs best OOD across all datasets achieving 8x to 28 x lower NRMSE OOD errors respectively in the 3 datasets than the best baseline. In Appendix C.I, we show that MetaPhysiCa learns the ground truth ODE (possibly reparameterized) for all dynamical systems.
6 CONCLUSIONS
In this work we considered the out-of-disrribttion (OOD) task of forecasting a dynamical system (ODE) under new initial conditions. We showed that existing PIML. methods do not perform well in these tasks and proposed MetaPhysiCa that uses a meta-learning framework to learn the causal structure for the shared dynamics across all environments, while adapting the task-specific parameters. Results on three OOD (initial condition) forecasting tasks show that MetaPhysiCa is more robust with 8x to 35x reduction in OOD eтror compared to the best competing baseline.
Limitations & future work. We believe that forecasting models should be robust to OOD shifts, and that our work takes step in the right direction with several potential avenues for future research: (i) Partial differential equations (PDEs): Extending MetaPhysiCa to forecasting PDEs under OOD scenarios is an interesting extension that requires an expanded set of basis functions that includes differential operators (like the Laplace operator), and considering -oitt--distriuttinn boundary conditions. (ii) More expressive structural causal models (SCMs): Our experiment on complex ODE task (in Appendix C.5) suggests that MetaPhysiCa with a more expressive SCM that allows for composition of basis functions is able to forecast out-of-distribution better than competing baselines, but suffers from learning stiff ODEs due to the complexity of a 2-layer learnable basis function procedure. Better optimization techniques may help alleviate this problem.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1XXY:1107..02933, 2019.
Vivek $ Borkar. Stochastic approximation with time scales. Systems & Control Letters, 29(5): 291-294, 1997.
Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data augmentation for neural pde solvers. arXiv preprint WwXiv:2202.076433 2022a.
Johannes Brandstetter, Daniel E Worrall, and Max Welling. Message passing neural pde solvers, In International Conference on Learning Representations, 2022b.
Steven L. Brunton, Joshua L.. Proctor, J. Nathan Kutz, and William Bialek. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings the National Academy of Sciences of the United States of America, 113(15):3932-3337, 2016. ISSN 10916490. doi: 10. 10.1073/pnas. 10073//aas.5113341133 1517384113.
Rick Chartrand. Numerical differentiation of noisy, nonsmooth data. International Scholarly Research Notices, 2011, 2011.
Gang Chen. Learning symbolic expressions via gumbel-max equation learner networks. arXiv preprint arXiv:2012.06921, 2020.
Gang Chen. Learning Symbolic Expressions via Gumbel-Max Equation Learner Networks. arXiv:2012.06921 [cs], May 2021.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6572-6583, Red Hook, NY, USA, December 2018. Curran Associates Inc.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neurul Information Processing Systems, 34: 25294-25307, 2021.
Matthieu Courbariaux, Yoshua Bengio, and -Perre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ I 2016.
or-1, arXiv preprint
M. Cranmer, Alvaro Discovering Symbolic Models from Deep Learning with Inductive Biases. NeurIPS, 2020a.
P. Battaglia, Rui Xu, K. Cranmer, D. Spergel, and S. Ho.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020b.
Yuchen Dang, Zheyuan Hu, Miles Cranmer, Michael Eickenberg, and Shirley Ho. Tnt: Vision transformer for turbulence simulations. arXiv preprint arXiv:2207.046166 2022.
Arka Daw, Anuj Karpatne, William Watkins, Jordan Read, and Vipin Kumar. Physics-guided neural networks (pgnn): An application in lake temperature modeling. arXiv preprint arXiv: 2017.
James H Faghmous and Vipin Kumar. A big data guide to understanding climate change: The case for theory-guided data science. Big data, 203):155-163. 2014.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Leaming for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1 1126-118 1135. PMLR, July 2017.
10

---

Under review as a conference paper at ICLR 2023
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivari- ant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, pp. 5318-3328. PMLR, 2021.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning In International Conference Machine Learning, pp. 1568-1577. PMLR, 2018
Robert Geirhos, Jöm-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1
2020.
AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning in linear systems. Advances in neural information processing systems, 31, 2018.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Christina Heinze-Deml, Marloes H Maathuis, and Nicolai Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application, 5:371-391, 2018.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9): 5149-5169, 2021.
Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour. Generalized Score Funetions for Causal Discovery. KDD proceedings. International Conference on Knowledge Discovery & Data Mining, 2018:1551 -1665 August 2018. ISSN 2154-817X. doi: 10.1145/
3219819.3220104.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference computer vision, pp. 1501-1510, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference on Learning Representations, 2017.
Chiyu "Max" " Jiang, Karthik Kashinath, Prabhat, and Philip Marcus. Enforcing Physical Constraints in Neural Neural Networks through Differentiable PDE Layer. September 2019.
Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing New Physical Systems via Context-Informed Dynamics Model. In Proceedings of the 39th International Conference on Machine Learning, pp. 11283-11301. PMLR, June 2022.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang. Michael Brenner, and Stephan Hoyer. Machine learning-accelerenn accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548-26566 2021.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Raphael Leiteritz and Dirk Pflüger. How Avoid Trivial Solutions in Networks. arXiv:2112.0m 12.05220 [cs, stat), December 2021.
Informed Neural
Julia Ling, Andrew Kurzawski, and Jeremy Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155-1666 November 2016. ISSN 0022-1 122-1120 120, 1469-7645. doi: 10.1017/jfm. 201.ff6....66655..
11

---

Under review as a conference paper at ICLR 2023
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):1-10, 2018.
Georg Martius and Christoph H. Lampert. Extrapolation and learning equations. arXiv:1610.02995 [cs], October 2016.
Viraj Mehta, Ian Char, Willie Neiswanger, Youngseog Chung, Andrew Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Neural dynamical systems: Balancing structure and flexibility in physical prediction. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 3735-3742, IEEE, 2021.
Ignavier Ng. Shengyu Zhu, Zhuangyan Fang, Haoyang Li, Zhitang Chen, and Jun Wang. Masked gradient-based causal structure learning. In Proceedings the 2022 SIAM International Conference on Data Mining (SDM), pp. 424 432. SIAM, 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
Ronan Perry, Julius von Kügglgeen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. arXiv preprint 220003200..00013, 2022.
Jonas Peters, Stefan Bauer, and Niklas Pfister. Causal models for dynamical systems. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 671-690. 2022.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932 955, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711. 2017a.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations. arXiv preprint arxiv: 17X.v: 10566, 2017b.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for data- driven discovery of nonlinear dynamical systems. arXiv preprint arXiv: /01102/00..0/266, 2018.
Paul K Rubenstein, Stephan Bongers, Bernhard Schölkopf, and Joris Mooij. From deterministic odes to dynamic structural causal models. arXiv preprint a6Xiv:6608..m (660.0800888 2016.
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-266 1992.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 2009.
Sebastian Thrun and Lorien Pratt. Learning learn: Introduction and overview. In Learning learn, pp. 3-17. Springer, 1998.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1457-1466, 2020a.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv: 2020b.
Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging thysics-based and data-driven modeling for learning dynamical systems. In Learning for Dynamics and Control, pp. 385-398. PMLR, 202
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. arXiv preprint 102 2021b.
12

---

Under review as a conference paper at ICLR 2023
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-ifformed neural networks. arXiv preprint arXiv:2203.07404, 2022.
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating physics- based modeling with machine learning: survey. arXiv preprint arXiv:2003 2020.
SHI Xingjian, Zhourong Chen, Hao Wang, Dan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-811 212-810,24 2015.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolci Du Ken-Ichi Kawarabayashi, and Stefanic Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations, 2021.
Alireza Yazdani, Lu Lu, Maziar Raissi, and George Em Karniadakis. Systems biology informed deep learning for inferring parameters and hidden dynamics. PLoS computational biology. 16(11): c1007575, 2020.
Kyongmin Yeo and Igor Melnyk. Deep learning algorithm for data-driven simulation of noisy dynamical system. Journal of Computational Physics, 376:1212-1231, January 2019. ISSN 00219991. doi: 20.10161.epp.211.100000 10.024
Yuan Yin, Le Vincent, DONA Jérémie, Emmanuel de Bezenac, Ibrahim Ayed, Nicolas Thome, et al. Augmenting physical models with deep networks for complex dynamics forecasting. In International Conference Learning Representations, 2021.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced inffrces neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823, 2022.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
13

---

Under review as a conference paper at ICLR 2023
Supplementary Material of o "MetaPhysiCa: Causality-aware Robustness to OOD Initial Conditions in Physics-informed Machine Learning"
A DESCRIPTION OF TASKS
For each dynamical system, we simulate the respective ODE to generate M 1000 training tasks each observed over regularly-spaced discrete time steps (fa...., (r) where vl, 4/2 0.11. Our data generation process is succinctly depicted in Table 1. For each dataset, the second column shows the state variables X, and the unknown parameters W For each training task T 1...., M,
sample an initial condition x(f) P(X,, F = e) where E = e is the training environment (shown under ID columns of the table). We consider two scenarios for the dynamical system parameters:
to
Constant W (third and fourth columns in Table 1): W(0+ is constant for all tasks i
For all tasks i, ( W puram as indicated in the table.
Varying (+)+ (final two columns in Table 1): We sample different W
U(W param: 2W param) for each task i with W param shown in the table.
At OOD test, we generate M' = 200 test tasks by simulating the respective dynamical system over timesteps (fa.-..,t,, ,,), where again Vl, 4 0.1l. For each test task j 1,..., M', we sample initial conditions X! ~ P(X,, E = e') where E = e' is the test environment and can induce a completely different support for the initial conditions x than in training. The distribution of the dynamical system parameters W is kept the same across training and test.
Damped pendulum system (Yin et al., 2021). The state X; = [0,, we] E R describes the angle made by the pendulum with the vertical and the corresponding angular velocity at time t. The true (unknown) function t describing this dynamical system is given by do, Wt, di -a sin(@:)- prwe where W = (a", , p²) ) are the dynamical system parameters. We simulate the ODE over time steps (fo:-.. -00....,yy (r) with f 0.17, T 100 in training and over time steps (fo,..., , t, } in test with r r= TT. In training, the pendulum is dropped from initial angles of? ~ ~((0 π x/2) with no angular velocity, whereas in OOD test, the pendulum is dropped from initial angles e U(x 0.1, r)) and angular velocity wto (j)
()),,
Predator-par prey system (Wang et al., 2021a). We wish to model the dynamics between two species acting as prey and predator respectively. We adapt the experiment by Wang et al. (2021a) to our out-of- distribution forecasting scenario according to Definition 1. Let and denote the prey and predator populations respectively. The ordinary differential equations describing the dynamical system is given by de = aap- p- p7, dg = 8* pg --" where W (,",,", ",,""""'"" are the (unknown) dynamical at
system parameters. We simulate the ODE over time steps {to.... .a.....,tty .tr with Vl, 0.11, = 100 in training and over time steps {to.. ... , tr) in test with T.. T. We generate M = 1000 training tasks
with different initial prey and predator populations with prey Pto u(1000, 2000) and predator
(6)
(6) 4to (10, 20) for each 1, ...., M. At OOD test, we generate M' 200 out-of-distribution
(OOD) test tasks with different initial prey populations pia ~ U(100, 200) but the same distribution for predator population dea ~ ((10,20)
(i)
Epidemic modeling (Wang al., 2021a). We adapt the experiment by Wang et al. (2021a) to our out-of-distribution forecasting scenario according to Definition 1. The state of the dynamical system described by three variables: number of susceptible (S), infected (1) and recovered (It) individuals.
df The dynamics is described using the following ODEs: ds = -3St N z
N 71, dR 71, where W = 7) are the (unknown) dynamical system parameters and - total population. We simulate the ODE over time steps (to..... tr } with VI, = 0.11, = 100 in training and over time steps = J.T. We generate M = 1000 training tasks with different initial populations for susceptible (S) and infected (7) individuals, while the number of initial recovered
is the
14

---

Under review as a conference paper at ICLR 2023
Test Normalized RMSE (NRMSE)
Constant W
Varying W
Methods
ID
OOD
ID
0OD
Standard Deep Learning
NeuralODE (Chen al., 2018)
0.003 (0.000)
1.229
0.059)
0.005 (0.000)
1.139
0.031)
DyAd (Wang et al., 2021b)
0.003 (0.001)
121.236
(462.022)
0.006 (0.001)
77.377
(204.435)
CoDA (Kirchmeyer et al., 2022)
0.004 (0.001)
2.044 0.754)
0.004 (0.001)
3.341
0.389)
Physics-informed Machine Learning
APHYNITY (Yin et al., 2021)
0.075 (0.071)
2.345 ( 6.035)
0.151
(0.150)
0.544
0.249)
SINDY (Brunton et al., 2016)
2.038 (0.008)
2.447
0.065)
1,999 (0.046)
2.746
0.476)
EQL (Martius & Lampert, 2016)
NaN*
NaN"
NaN"
NaN"
MetaPhysiCa (Ours)
0.006 (0.002)
0.035
0.028)
0.009 (0.004)
0.019
0.002)
(a) Test NRMSE | for different methods. NaN indicates that the model returned errors during test.
MetaPhysiCa
NeuraIDDE
Aphynity
SINDY
DyAd
EQL
True
10.0
30.0
75
75
5.0
25
25
75
(b) In-distribution predictions
(c) OOD predictions
Figure 5: (Epidemic model results) (a) MetaPhysiCa outputs 35x and 28x robust OOD predictions in constant W (+) and varying ((+) datasets respectively. (b, shows example ground truth curves (blue stars) in- and out-of-distribttion along with corresponding predictions. Only MetaPhysiCa (orange) closely follows the OOD.
variahles
Constant W
panzn
Varying
W6
24 paman 2W paunn
Datasets
ID
ODD
ID
0OD
X, (8,,2)
6o ~ ((000=/2)
Og 21(T 0.1,z))
8, ~ 2410, m/2)
Ag ~ U(( - 0.1,
Damped pendulum
40
L/ 1. Apuzan 0.2
A0 ((
W (o.p)
Opana
X. (p)
~ 24(1000, 2000)
Po ~ ((100 200)
Do 10000 2000)
Po ~ 24(00 200)
Predator prey system
8o 24(10, 20)
9u 2110
2o L(110, 20)
do ((1,220)
Opennnn
1, 1aaa = 0.06, 7pmmm
0.5, 4aan 0.0005
So 7/(9. 10)
So (00, 100) (1,55 Rg 0
So ~ 2/, 10)
So ~ (900 100) (1.5))
(Sc, Rc)
5)
Jo 2(1155
Epidemic modeling
Ro = 0
Ro
R. =0
(8,5)
8pam
Table Description of the dataset generation process. For each dataset, X, denotes the state variable of the dynamical system and W denotes its parameters. Third and fourth columns correspond to the case when the (hidden) ground truth parameters W are kept fixed for all the tasks to ( = Wprram- For example, in the damped pendulum dataset, we fix 3(0) = Spuram = and p(+) = Pauram = 0.2 for all tasks i. Column ID represents in-distribution initial states while the column OOD represents the out-of-distribution initial states. Similarly, the final two columns correspond to the case when the ground truth parameters ( vary across tasks and are sampled from uniform distribution (i)+ ~ U(W param: 2W param) For example, in the damped pendulum dataset, we sample (1)= ~ 21 (Apaaram 20 20raamm)) (1, 2) and pitt* ~ [ 2ppuram) (0.2, 0.4) for each task i.
(R) individuals are always zero. In training, we sample 5( ~ (9, 10) and T(00) ~ (1155) for each i = I, .,.... M. At OOD test, we generate M' 200 out-of-distribution test tasks with different initial susceptible population, ا(ي) ~ U(90, 100), while keeping the same distribution for infected population.
15

---

Under review as a conference paper at ICLR 2023
FCC (0
:4) RM"
+
1
With
D
sin(X:;)
Training
W
doom
Training Loss
Predicted derivaaiivs time
Tiret
O 1
Equation (3)
Shared
0
tos by
or Equation (6)
Iruth derivatives for time Eop...
{0, 13
dxm
3
+
W(M)
Raom
F(X MM
:0 EX
Time
Task-specific Basis functions parameters
Trajectories from unknown dynamical system
Causal structure
Observations from time
Observe
Predict
Test
101
x sarny
ODE Solve
Test-time adant Equation
110
sin 2
& E {0, (0,1)dmm
((++1)
Rdxm
(Fixed at test)
F(X.
(6) e R2"
Predict test Irajectory till by with parameters c and gg usiin an ODE solver.
Figure 6: Schematic diagram of MetaPhysiCa and corresponding training/test methodologies. We observe M trajectories in training from the same dynamical system with different initial conditions and ODE parameters.
training, 6 denoting the causal structure, is shared among all tasks i I, ,,..,, M, while (i) are the task-specific parameters. Predicted derivatives for task over time 1 lo lo...,, lr are obtained from Equation (2) using the parameters 4, ((( and the basis functions F(X (0) E). During test, we adapt (M++1) over the observations of the test trajectory from time
la, ly. keeping the learnt causal structure & fixed.
B IMPLEMENTATION DETAILS
In what follows, we describe implementation details of MetaPhysiCa and the baselines.
B.1 METAPHYSICA
Figure 6 shows schematic diagram of MetaPhysiCa and the corresponding training/test procedures. Recall from Equation (2) that the proposed model defined as
dx (i)
(W o ())
(5)
where is the Hadamard product and
F(X(" = ( tions with parameters g,
fm (x Em) {m)
is the vector of outputs from the basis func-
(0, 1)
1 jdxm are the learnable parameters governing the global causal structure across tasks iff edge 2,,1 - dæi./dt exists,
such that 4
w e Rdxm are task-specific parameters that act as coefficients in linear combination of the selected basis functions.
In our experiments, we use polynomial and trigonometric basis functions, such that
F(X (
(02 L1
sin(E1, 11 + E1.2 sin(Ea, x + Ea,2)
d.d
polynomial onder
polynomial order
triganametric
16

---

Under review as a conference paper at ICLR 2023
Equation (3) describes a bi-level objective that optimizes the structure parameters & and the global parameters Ç in the outer-level, and the task fask-specific parameters W in the inner-level as follows
$ É arg min 2
R
P. E)
Ao |1 ARES Variance (( (W(1, D ()E))
M
i=1
s.t. W
arg min R (W (0) +.E) + Aw 11,
Vi 1..... M
w
where Ag, Aw and AREx are hyperparameters. As discussed in the main text, the jointly optimizing P f and W .....: M, instead of alternating SGD resulted in comparable performance with considerable computational benefits. We use the following joint optimization objective to approximate Equation (3),
M
$ c. (1)
WMM
arg min
R 1 (W
Φ. + oo 1
Aw
W
4. (
W(M)
M
(6)
XEE Variance (1)) (W Φ, {))=))
We perform grid search over the following hyperparameters: regularization strengths As
10-3), 10-3,10 and learning rates E {10-2, 10 10 We choose the hyperparameters that result in sparsest model (i.e.,
{10-1, 1,0--.5x000 10 10 10-21, Aw E {0, 10-1, 11-1011000,100011111111111 10,110--,,, ARES {0, 10-3,
with the least [l)] while achieving validation loss within 5% of the best validation loss in held-out in-distribution validation data.
B.2 NEURALODE (CHEN ET AL., 2018)
The prediction dynamics corresponding to the latent NeuralODE model is given by 4x or Fm (X,, Zgr Wi) where Σgr Fene (x.,..., X,,: W2) encodes the initial observations using a recurrent neural network Fene (e.g., GRU), and Fm is a feedforward neural network. The model trained with an ODE solver (dopri5) and the gradients computed using the adjoint method (Chen et al., 2018). We perform a grid search over the following hyperparameters: number of layers for Fm. 21, 2, 3). size of each hidden layer of Fans dn {32, 64, 64, 128), size of the encoder representation Zgr, d- {32, 64, 128]. batch sizes B E (32, 64), and learning rates 17 E (10-2, 10-3, 10-1).
B.3 DYAD (MODIFIED FOR ODEs) (WANG ET AL., 2021B)
DyAd, originally proposed for forecasting PDEs, uses meta-learning framework to adapt to different training tasks by learning per-task weak label. We modify their approach for our ODE- based experiments. Since we do not assume the presence of weak labels for supervision for adaptation, mean of each variable in the training task as the task's weak label. We NeuralODE as the base sequence model for the forecaster network. The forecaster network takes the initial observations as input and forecasts the future observations while being adapted with the encoder network. The encoder network is a recurrent network (GRU in our experiments) that takes as input the initial observations and predicts the weak label. The last layer representation from the encoder network is used to adapt NeuralODE via AdaIN (Huang & Belongie, 2017). We perform grid search over the following hyperparameters: size of hidden layers for the forecaster and encoder networks da E (32, 64, 128), number of layers for the forecaster network, I. E 21,,,3), batch sizes B E (32, 64), and learning rates r) E (10-2, 10-3, 10-1).
B.4 APHYNITY (YIN ET AL., 2021)
given (possibly incomplete) physics model d(,, Ophy) with parameters Ophy- When the training data may consist of tasks with different (i),, APHYNITY predicts the physics parameters with respect to the task i inductively using a recurrent neural network Gmn from the initial observations of the system ( Gun (X Xx,;; W2). Then, phy
APHYNITY assumes that
17

---

Under review as a conference paper at ICLR 2023
APHYNITY augments the given physics model with feedforward neural network component Fan and defines the final dynamics dx Φ X ( θ phy Fm ( W1). APHYNITY solves constrained optimization problem minimize the norm of the neural network component while still
predicting the training trajectories accurately. The model is trained with an ODE solver (dopri5) and
the gradients computed using the adjoint method (Chen et al., 2018). In our experiments, provide APHYNITY with simpler physics models:
For damped pendulum system, we use a physics model that assumes no friction: -a sin( where phy = appyy is the physics model parameter.
de dt
Wes
duls at
For predator-prey system, we use a physics model that assumes no interaction between the two
dp = Oppyy, dq = 7phyy where pphy = (Aphy, "pby)) are the physics model parameters. species: de
dt
For epidemic model,
physics model that assumes the disease is not infectious: ds I, where Pphy = Yphy is the physics model parameter.
0, de T
In each dataset, A APHYNITY needs to augment the physics model with neural network component for accurate predictions.
We perform grid search over the following hyperparameters: number of layers for Fum- L E 21, 3) size of each hidden layer Pons dn. {32, 64, 128), batch sizes B E {32, 64), and learning rates E {10-2,10' 11-0,110111 10-2).
B.5 SINDY (BRUNTON ET AL., 2016)
SINDy uses given dictionary of basis functions to model the dynamics as 2x O ( W where θ is feature map with the basis functions (such as polynomial and trigonometric functions) and W is simply weight matrix. SINDy trained using sequential threshold least squares (STLS) for sparse weights W. We perform grid search over the following hyperparameters: threshold parameter used STLS optimization, To E {0.005, 0.01, 0.05, 0.1, 0.2, 0.5), and the regularization strength (0.05, 0.01, 0.1, 0.5..
di
B.6 EQUATION LEARNER (MARTIUS & LAMPERT, 2016)
Equation learner (EQL) is neural network architecture where each layer is defined as follows with input æ and output
z Wx bb
(fi(=))) (I(,,, #x41), 02(=x22 2k43).....).
where Si are unary basis functions (such sin, cos, etc.) and Ji binary basis functions (such as multiplication.. We use id, sin and multiplication functions in our implementation. EQL trained using sparsity inducing é-regularizztion with hard thresholding for the final few epochs. We perform a grid search over the following hyperparameters: number of EQL lay- ers, L E {1,2]. number of nodes for each type of basis function, h 31 5}. regulariza- tion strength a {10-1, 10 10100 10 1010 10 "), batch sizes B {32, 64), and learning rates n E (10-2, 10-30-300 10-1).
C ADDITIONAL RESULTS
C.1 QUALITATIVE ANALYSIS
Recall from Equation (2) that the proposed model is defined as
dx
(w @ +) F(F(F( ()
dt
where F(X()) {) is the vector of outputs from the basis functions, E {0, 1 are the learnable parameters governing the global causal structure across all tasks, and (i e R are tpsk-specific parameters that act coefficients in linear combination of the selected basis functions.
18

---

Under review as a conference paper at ICLR 2023
Dalasets
State variables
Ground Iruth ODE
Learnt ODE (from bb)
d6,
d0,
Wiwh
Damped pendulum
X; (4)
a²² sin(0.) - p wi
Aus
W2 sin(0r) + Wawn
a
or
Predator prey system
X, (pp)9t)
de a' pr B¹ B'Pe
dp.
Wipe + Wapiq:
du 8 pes -7-4
4
WaPin: Wag:
d
d8,
3
S.T.
45.
WiS:
dE
3.1. 1
AE
Epidemic modeling
X, = (St, Jr. R()
dL 8 BITL RG -4
dA
W2S,T, + Wa + W.
Y" I
4R. W3S.4 + Wol? + W,I..
Table 2:
(Qualitative analysis.) Ground truth dynamical system vs learnt ODE in the meta-model 4. Recall that & E {0, ]dxm dictates which of the basis functions affect the output dXt/dt. The weights W, in the learnt ODE column are learnable parameters that are optimized via test-time adaptation in Equation (4). MetaPhysiCa learns the exact ground truth ODE for Damped pendulum and Predator-pin system, and reparameterized version of the true ODE for epidemic modeling task.
After training, the ODE learnt by the model can be easily inferred by checking all the terms in & that are greater than zero, i.c., 2 > 0 implies fü (2;;; Ex) + das.sftt exists in the causal graph. In other words, RHS of learnt ODE for das.s/dt contains the basis function ;
Table 2 shows the ground truth ODE and the learnt ODE for the three experiments. For each learnt ODE, we also depict the learnable parameters Wi that can be adapted using Equation (4) during test-time. For damped pendulum and predator-prey system, the RHS terms in the learnt ODE exactly matches ground truth ODE, and from Figures and 4, clear that the method is able to accurately adapt the learnable parameters Wi during test-time. For epidemic modeling task, MetaPhysiCa learns a reparameterized version of the ground truth ODE. For example, MetaPhysiCa dR,
dRt learns di W I.S. + W + W.,,,, which can be written as di Wal (the ground truth ODE) if W W W because S, + It + R. N is constant denoting the total population. While the learnt reparameterized ODE is more complex because allows different values for W W W the test-time adaptation of these learnable parameters with the initial test observations results in them taking the same values.
C.2 ABLATION RESULTS
We present an ablation study comparing different components of MetaPhysiCa in Table 3. Table shows -uitof-disti test NRMSE for MetaPhysiCa without each individual component on the three dynamical systems (varying W scenario). We observe that sparsity regularization (i.e., [[ and test-time adaptation are the most important components. For two out of three tasks, the method returns prediction errors without sparsity regularization.
When testing MetaPhysiCa without test-time adaptation, we simply use the mean of the task-specific weights learnt for training tasks as the task-specific weight for the given test trajectory, i.e, W+++1 = I M Σ, ( This results high OOD errors showing the importance of test-time adaptation. The other two components of the MetaPhysiCa, the task-specific é éyrreuularization (i.e., (/..|/ and the V-REx penalty (Krueger et al., 2021) help in some experiments and perform comparably others.
C.3 OUT-OF-DISTRIBUTIIO ODE PARAMETERS
The forecasting task in Definition 1 considers -ut-of-distribution initial conditions X,, and in- distribution ODE parameters W Here, we consider OOD values for true dynamical system parameters (i)+ as well, which significantly increases the difficulty of the forecasting task.
Consider the damped pendulum system: de, we, dt
dw: 2
2 sin ())-))p- Wi where W = (a",p")
are the dynamical system parameters. Training: Pendulum is dropped from initial angles 8(9) ~ 4(00, a/2) with angular velocity. We sample the dynamical system parameters aft) 2(1122
19

---

Under review as a conference paper at ICLR 2023
OOD Test Normalized RMSE
Method
Damped Pendulum
Predator-Prey
Epidemic Modeling
MetaPhysiCa without ||0|]] |
0.070 (0.011) NaN*
0.129 (0.030)
0.019 (0.002)
1.806 (0.736)
NaN
without [[
0.434 (0,531)
0.132 (0.020)
0.020 (0.021)
without test-time adaptation
1.223 (0.741)
1.404 (3.794)
0.358 (0.554)
without V-REx penalty
0.070 (0.014)
0.129 (0.030)
0.042 (0.065)
Table 3: (Ablation.) Out-of-distribution test NRMSE for MetaPhysiCa without each individual component on the three dynamical systems (varying W scenario). Sparsity regularization (i.e., [1 and test-time adaptation are the most important components, whereas the task-specific & d-rrgularriation (i.e., [[W[]] and the V-REx penalty (Krueger et al., 2021) help in some tasks, but not in others.
Test NRMSE $
Methods
ID
OOD X. OOD X and W
Standard Deep Learning
NeuralODE (Chen et al., 2018)
0.083 (0.033)
0,591 (0.119)
1.208 (0.401)
DyAd (Wang et al., 2021b)
0.078 (0.051)
0.834 (0.263)
1.390 (0.441)
CoDA (Kirchmeyer et al., 2022)
0.052 (0.032)
0.764 (0.201)
1.031 (0.213)
Physics-informed Machine Learning
APHYNITY (Yin et al., 2021)
0.097 (0.020) NaN*
0.970 (0.384)
1.343 (0.404) NaN
SINDY (Brunton et al., 2016)
NaN
EQL (Martius & Lampert, 2016)
NaN*
NaN
NaN
MetaPhysiCa (ours)
0.049 (0.002)
0.070 (0.011)
0.181 (0.012)
Table 4: (Damped pendulum.) Normalized RMSE 1. of test predictions from different methods under two cases: (a) when initial conditions Xz, are OOD, and (b) when both initial conditions X and ODE parameters W are OOD. NaN* indicates that the model returned errors during test-time predictions. MetaPhysiCa is able to adapt its parameters to the OOD parameters Wi and outputs z 5x more robust OOD predictions compared to the baselines.
and (00) (00.2, 0.4) for each task i. Out-of-disttibuttio Test: The pendulum is dropped from initial angles ((6) U(T 0.1, IT) and angular velocity w((h E Ed(-1,0) 0). We sample the dynamical system parameters a (+) ~ ((1,2) and p(X) ~ 0.2),. Note that the damping coefficient p(*) sampled out-of-support from its training distribution. Rest of the experimental methodology is kept same as before.
We report the normalized RMSE of the all the methods in Table 4 for three test scenarios: in- initial
distribution (ID), conditions and ODE parameters (OOD X and (f))). MetaPhysiCa is able to adapt relatively well to the out-of-distribution ODE parameters and performs 4x better than the best baseline. Unfortunately, the test-time adaptation is not perfect (NRMSE is 5x higher for OOD X and W(()+ compared OOD initial conditions alone), possibly because the trajectories with higher (6) and higher (6) are harder to forecast.
initial conditions (OOD X and out-of-distributie
C.4 ROBUSTNESS TO NOISE
We repeat the Damped pendulum and Predator-pres prey experiments with increasing amounts of noises. Specifically, we add 1%, 5% and 10% Gaussian noise to all the trajectories, both in training and in test. As discussed before, we use Total Variation Regularization (TVR) (Rudin et al., 1992; Chartrand, 2011) for estimating derivatives from noisy data as done by Brunton et al. (2016). We report the normalized RMSE for different models trained the noisy versions of data in Figure 7. SINDy and EQL are not shown as they returned errors during test-tine time predictions similar to the case with no noise because the learnt ODE was too stiff (numerically unstable) solve. In both tasks, the proposed
20

---

Under review as a conference paper at ICLR 2023
1.75
NevstaDt
ReuraGDe
1.50
Ryed
APHINTY
2.0
RyAc APHNVIT
1.25
Mauutya
1.5
NRMSE
1.00
NRMSE
0.75
1.0
0.50
0.25
0.5
0.00
0.0
10
Percentage (%) of Gaussian noise
Percentage (%) of Gaussian noise
(a) Damped pendulum task
(b) Predator-prey system
Figure 7: (Performance with increasing noise.) cuttof-distributtio NRMSE values for Damped Pendulum and Predator-pres preo experiments with different percentages of Gaussian noise added (0%, 1%, 5%, 10%). MetaPhysiCa is relatively robust to S 5% Gaussian noise and outper- forms the baselines. With a larger amount noise, MetaPhysiCa is unable to identify the dynamical system accurately but performs comparable to the baselines.
method is relatively robust to small amounts of noise and outperforms the baselines. With 10% noise, MetaPhysiCa unable to identify the dynamical system accurately, but performs comparable the baselines.
C.5 COMPLEX ODE TASK
In this section, we extend MetaPhysiCa to consider significantly more expressive structural causal models (compared to Figure 3) that allow for composition of the basis functions. This is achieved with a 2-layer learnable basis function composition procedure. For example, given basis func- tions fi (1) sin E1,2), and (2());; Øt, 11,, one can construct more ex- sin sin(Es,1 + E3,2) S3,4), [1(m;;; (4) = æt, 22,2 sin($4, 22,1 + E4,2), etc., where Ej are global parameters that remain con- stant for all training/test tasks. The rest of the SCM remains the same and the derivative dx dt for particular dimension j E (I....,d d} is a sparse linear combination of the original basis functions and the more expressive second layer ones.
pressive basis functions with compositions: fs((;;;))
We evaluated MetaPhysiCa on a more complex ODE task from Chen (2020) adapted our setting. We consider two-dimensional ODE with state X, = [pt, q:] E _2: dps = a sin(p:)+ + sin(g?); des de e" sin(p:) cos(g:). where W (a" 6 ct) are the dynamical system parameters. We simulate the ODE over time steps (tg.... ty with VI, t = 0.11, T = 100 training and over time steps .T In training, we sample initial states Pr, qr (0(.5, 1), whereas in (to.... ......... fr) } in test with dii-t---istiibution test, we sample Pt, 9: ~ (111 1.5). For constant ( scenario, the dynamical system parameters are set to (+) = b(3) c(-- = for i, whereas for the varying W((+ scenario, the dynamical system parameters are sampled as (+))+ 5(6). cfi) ~ 2.(110, 1.5).
de
Table 5 shows the results for this task. First, we note that due to the complexity of 2-layer learnable basis function procedure, we sometimes need use validation data (held from training) cross- validate the learned model (and reject meta- meta-models that do not do well in validation). MetaPhysiCa learnt a stiff ODE for out of 5 folds of cross-vale validsaai,,,,nn, resulting no predictions for in-distribution validation data, which were rejected (marked as superscript +). In these experiments MetaPhysiCa performs 1.5x to 1.7 better than the competing baselines. We believe there room for improvement in the optimization procedure of these more complex models.
21

---

Under review as a conference paper at ICLR 2023
Test Normalized RMSE (NRMSE)
Constant W
Varying W((+
Methods
ID
OOD
ID
OOD
NeuralODE (Chen et al., 2018)
0.012 (0.001)
0.188 (0.025)
0.034 (0.008)
0.296 (0.064)
APHYNITY (Yin et al., 2021)
0.010 (0.002)
0.329 (0.050)
0.027 (0.010)
0.684 (0.117)
SINDy (Brunton et al., 2016)
NaN"
NaN
NaN*
NaN
MetaPhysiCa (Ours)
0.119 (0.072)*
0.110 (0.048)*
0.188 (0.035)*
0.203 (0.046)* *
Table 5: Test NRMSE for different methods. indicates that the method returned errors during predictions due learning stiff ODE.
22


Reviews:
[{"review_id": "psRXwBJaO5", "reviewer": "Reviewer_s2pT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the challenging problem of learning robust solvers for ODEs in physics-inspired machine learning, mainly when initial conditions at the test time are the out-of-distribution. First, with simple experiments on well-motivated simulated physics tasks (such as damped pendulum systems and predator-prey), the paper presents the limitations of existing PIML algorithms in dealing with OOD initial conditions. Next, the authors propose meta-learning algorithms that embed a structural causal model to identify the most suitable model for the ODE task to tackle this. Finally, evaluating the normalized RMSE, experiments across multiple domains suggest that MetaPhysiCa is competitive on in-distribution tasks but significantly outperforms the baselines in OOD settings. ", "summary_of_the_review": "The paper studies challenges in forecasting dynamical systems with OOD initial conditions and the limits of current PIML-based approaches. Building on insights from causal inference and multi-task learning, the authors propose MetaPhysica. This hybrid algorithm performs a structural causal search on a family of SCMs (under realizability) to perform appropriate interventions. With extensive experiments and comparisons to relevant baselines, the proposed algorithm outperforms prior work on OOD tasks by a considerable margin. ", "strength_and_weaknesses": "### **Strength**\n+ The paper is well-written, with pedagogical examples highlighting the limits and challenges of current PIML algorithms. The authors do a great job presenting their contributions in the context of existing literature. \n+ By effectively combining the strengths of meta-learning algorithms and causal inference, the proposed algorithm significantly improves performance on OOD tasks in well-studied benchmark physics simulation tasks (such as damped pendulum and predator-prey).\n\n### **Weakness**\n- The assumptions of realizability in the family of structural causal models seem restrictive and might not hold in more complex ODE tasks. A discussion on the scope of these assumptions would significantly help understand the algorithm's limitations and strengths. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WWKqUtOEm2", "reviewer": "Reviewer_t2KB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Current deep and physics-informed machine learning models struggle to correctly forecast dynamical systems in out-of-distribution (OOD) settings. This paper proposes a new approach consisting of a meta-leaning strategy combined with causal structural discovery. The method is evaluated on standard ODE benchmarks, showing better forecasting performances on OOD initial conditions.", "summary_of_the_review": "In summary, this hybrid approach for OOD dynamical forecasting is appealing and shows promising results. However, in the current state of the paper, it is hard to really understand the working principle and if the method would apply to more complex physical systems.\n\nEDIT: after discussion with other reviewers, I have still concerns on the clarity of the method. The positioning with respect to Coda and DyAd is not discussed and  the reason that this method outperform the other by an order of magnitude is still unclear for me. Besides, I do not understand in experiments why Aphynity is largely inferior to the data-driven Neural ODE for the in-distribution pendulum. \nI return to my original pre-rebuttal score. ", "strength_and_weaknesses": "Strengths:\n\nThis paper is well-written and very enjoyable to read. It tackles a very relevant issue that undermines physics-informed machine learning in OOD settings. The method is clearly presented and well positioned with respect to the state-of-the-art. The combination of the structural causal model with meta-learning is an appealing idea. This method outperforms with a large margin other competitors on OOD initial conditions, while remaining equivalent in-distribution. \n\nWeaknesses:\n\nW1: The experimental section contains only forecasting performances in and out of distribution. Without an in-depth analysis of the model, for example an ablation study, it is hard to understand what elements make this model more successful. Could you provide (possibly in appendix) a more thorough analysis? For instance, what is the impact of the SCM? Have you analyzed the discovered causal structure and what interpretation can be drawn? \n\n\nW2: Moreover, considering the robustness to ODD initial conditions is fine and interesting, but why not addressing in the paper the robustness to ODE system parameters? This is done in CoDA and DyAd and I a wondering if this method would still have such a performance gain in this context. This would be interesting for the community to analyze what source of ODD interventions have the most impact.\n\nW3: The authors make experiments on common benchmark ODEs (damped pendulum, predator prey, epidemic). Why not evaluate your method on more complex PDE equations? Published methods evaluate on 2D reaction-diffusion and Navier Stockes (for CoDA) and turbulent flows and sea surface temperature (for DyAd). \n\nQuestions:\n\nQ1: The loss minimized in practice is a MSE between the derivatives along the trajectories. How is the derivative estimated for the ground-truth trajectory (finite differences)? Are these estimates noisy with noisy data and how does it affect performances? \n\nQ2: In Eq 3, the authors write the global optimization objective. Is this the real bi-level objective optimized in practice? An algorithm script could help to clarify the training scheme. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MhFOoDHzO5", "reviewer": "Reviewer_hnK8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method called MetaPhysiCa, which is a physics-informed machine learning method to better model dynamical ODE systems tested under OOD Initial conditions with changing or non-changing ODE parameters. The approach is making use of various meta-learning and causal structure discovery techniques in combination with a predefined set of possible basis functions to parametrize the RHS of an ODE. Authors validate their approach on 3 different ODE setups (damped pendulum, predator-prey and epidemic modelling) outperforming multiple baselines in OOD test scenarios.", "summary_of_the_review": "I think this paper is a nice contribution by proposing a new physics-informed ML method for ODEs that exhibit more robust generalization with respect to OOD initial conditions and makes use of several important concepts and necessary assumptions to achieve this. While I appreciate the number of experiments and convincing performance over existing baselines I have mentioned a few remaining concerns as the studied setups appear to be particularly easy for the here proposed method. I am concerned that the method might not be very applicable beyond such very simple ODEs that adhere to the structure assumed by the method. I, therefore, lean towards rejecting but I am willing to change my rating if the authors can address my concerns.\n", "strength_and_weaknesses": "Strengths:\n- The paper is very clearly written and the problem setup, as well as the proposed method, are presented comprehensively.\n- The authors validate their method on three prominent ODE tasks and benchmark their method against a multitude of relevant baselines\n- The paper touches upon an important problem setting and I appreciate the general idea of the method to obtain better OOD generalization via causality-inspired regularization and modelling assumptions.\n- Nice and convincing experimental results\n- Nice contextualization of the method with respect to prior works\n\nWeaknesses:\n- The method appears to make several rather strong assumptions on the underlying ODE system that make the method only apply to a smaller subset of ODE systems. (1) The underlying ODE system is assumed to be parametrized with the task-specific parameters W acting as coefficients in a linear combination of basis functions, which happens to be exactly the case for the studied toy experiments. I thus would expect the method to potentially exhibit several issues if this is not possible. The three environments are all building upon the sparsest possible causal structure in that every RHS is a single term such that the sparsity regularization will always help as long as exactly one entry per row in the structure parameter matrix survives. It would be important to understand how the method performs if there are linear combinations of more than one basis function in the RHS. Finding a good trade-off in the right regularization through $\\lambda_\\Phi$ and its effect on OOD generalization might quickly become very challenging. While I understand that the more general setting is much harder and I appreciate the progress over baselines in the here studied experiments I would have appreciated it if these limitations and assumptions are stated more prominently as it seems the method predominantly benefits a lot from being given optimal basis functions.\n- Presentation (minor): The plots in Figures 4 and 5 should be enlarged and could be improved as the labels and legends are barely readable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HPcgXahqwb_", "reviewer": "Reviewer_MowG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed MetaPhysiCa, a treatment to OOD initial conditions in PIML, via meta-learning algorithms with structural causal models.\n", "summary_of_the_review": "In summary, this is an interesting paper. I am willing to raise my score if the authors could conduct a more comprehensive investigation on the current scalability of the proposed approach. ", "strength_and_weaknesses": "Strength:\nI like the motivation of this paper, and the illustrative examples are effectively demonstrating the limitations of vanilla PIML.\n\nWeakness:\nI would prefer that the authors state clearly the current limitation of this approach. The SCM adopted in this paper would not scale to the realistic, complex ODE cases, and either some experimental investigation or discussions on the limit of this approach would be beneficial. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EM8vSxW19w", "reviewer": "Reviewer_rgk4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on dynamical system prediction in the case where the initial conditions of test trajectories are sampled differently from that of training. Metaphysica is introduced, a model based on the identification of causal models which are then finetuned on the new test trajectories to maintain good performance on the new OOD trajectories.", "summary_of_the_review": "The paper suffers from its poor writing, and its lack of experience and analysis to clearly justify the successes of MetaPhysiCa and the failures of other models. As it stands, I do not recommend this paper for acceptance. Nevertheless, the proposed method is interesting, as is the task, which seems to me to have potential.\n\n## EDIT\nThe authors provided very convincing answers to my questions. In my opinion, there are still two weaknesses to the proposed method:\n1) This is based on a dictionary of functions, and the model assumes that this contains the functions necessary to model the dynamics. This is a difficulty that is also found in SINDy, and many other methods.\n2) Experiments are performed on relatively simple dynamical systems. The contribution would have been reinforced if it could have been applied to more complex systems, even real ones, although I am aware of the difficulty of bringing together a dataset for this task.", "strength_and_weaknesses": "First, I would like to point out that the paper is, in my opinion, very poorly written. Almost all of the arguments are detailed in a convoluted way that is difficult to read, especially section 2 and 3. Figure 1 needs to be re-worked. It is difficult to separate contributions from their poor presentation, so my review may contain misinterpretations.\n\nMany points bother me about the method used. My main concern is on the adaptation phase: it seems that you train your model on the test data. This is a very dangerous choice, which requires a lot of carefulness about how to achieve it. Here are some leads:\n- The amount of data on which MetaPhysiCa is adapted in test-time is not so negligible, according to the appendix. The risk of overfitting is huge. I suggest using a validation set from the same distribution as the test set but different to adapt the model, and then test it on a completely unseen set of initial conditions.\n- It would also be interesting to see how the model reacts if it is again exposed to the training trajectories after adaptation. My fear is that MetaPhysiCa performs poorly, due to overfitting on testing data.\n- When MetaPhysiCa is trained on constant $W*$ tasks, the adaptation phase should be unnecessary. Its ablation in this configuration would confirm that the model is indeed able to identify the causal model.\n\nI strongly recommend that authors demonstrate indisputably that adaptation during test-time is reasonable and relevant.\n\nI also disagree with several claims:\n- I see many similarities between MetaPhysiCa and SINDy: the two models seek to identify an analytical model describing the dynamic system. I would have appreciated a much clearer discussion of the differences between MetaPhysiCa and this baseline. I particularly disagree with the sentence \"These transductive methods, however, do not transfer knowledge learned in training to predicting test examples unseen during training\". SINDy (and EQL to a lesser extent) makes it possible to identify an analytical dynamic equation from the data. This equation is (theoretically) general and therefore perfectly transferable to other initial conditions. The failure of SINDy seems very surprising to me.\n- Moreover, MetaPhysiCa seems to me to be able to (even forced to) identify the real dynamic equation. This is a simple check to perform. It would then interesting to verify that the causal graph is correctly identified, which would make it possible to generalize OOD. That being said, I don't see what prevents SINDy from doing the same, especially in the case where the $W*$ parameters are fixed. I would appreciate an in-depth analysis of this.\n- The authors justify the failure of methods like APHINITY or NODE by arguing that the neural networks are not algorithmically aligned to the problem. This is quite reasonable, but in my opinion makes the comparison unfair, since the basis functions necessary to solve the problem are directly implemented in MetaPhysiCa by hand. What happens if the structure of NODE is adapted to suit the problem?\n- The authors propose constrain their causal graph to be minimal. Although this is a common assumption for the identification of the equations of a dynamical system, a discussion on the relevance of this choice (and its interest in practice) seems to me necessary.\n\nFinally, section 5 shows results that I don't understand:\n- The total failure of SINDy and EQL seems very surprising to me. These methods identify an analytical equation which should therefore produce relatively correct results. Their failure requires further analysis.\n- The figures presented in addition to the tables show predictions from baselines and MetaPhysiCa. However, I don't understand how the initial conditions (ie the starting point of the curves) can be different from one baseline to another. Could it be that the baselines are evaluated on different trajectories? Authors must justify this.\n- I suspect the Deep learning baseline to overfit on the relatively simple system chosen by the authors. However, I could not find information about the size of the networks used for NODE and APHINITY. I believe that the OOD failure of this model maybe tempered with smaller models less subject to overfitting. I would appreciate if the authors could discuss this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject