Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Authors: Shurui Gui
Xiner Li
Shuiwang Ji
Venue: ICML 2025 poster (2025)
Keywords: ordinary differential equation
invariant learning
Primary area: deep_learning->sequential_models_time_series

Abstract: We consider learning underlying laws of dynamical systems governed by ordinary differential equations (ODE). A key challenge is how to discover intrinsic dynamics across multiple environments while circumventing environment-specific mechanisms. Unlike prior work, we tackle more complex environments where changes extend beyond function coefficients to entirely different function forms. For example, we demonstrate the discovery of ideal pendulum's natural motion $\alpha^2 \sin{\theta_t}$ by observing pendulum dynamics in different environments, such as the damped environment $\alpha^2 \sin(\theta_t) - \rho \omega_t$ and powered environment $\alpha^2 \sin(\theta_t) + \rho \frac{\omega_t}{\left|\omega_t\right|}$. Here, we formulate this problem as an *invariant function learning* task and propose a new method, known as **D**isentanglement of **I**nvariant **F**unctions (DIF), that is grounded in causal analysis. We propose a causal graph and design an encoder-decoder hypernetwork that explicitly disentangles invariant functions from environment-specific dynamics. The discovery of invariant functions is guaranteed by our information-based principle that enforces the independence between extracted invariant functions and environments. Quantitative comparisons with meta-learning and invariant learning baselines on three ODE systems demonstrate the effectiveness and efficiency of our method. Furthermore, symbolic regression explanation results highlight the ability of our framework to uncover intrinsic laws.

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Shurui Gui
Xiner Li!
Shuiwang Ji
Abstract
1999
Singh et al. 2012 Wang al. 2016 Lusch al. Yeo & Mclnyk 2019 Giannakis 2019 Numerous efforts (Kirchmeyer et al. 2022 Wang et al.) 2022 Mouli
We consider learning underlying laws of dynam- ical systems governed by ordinary differential equations (ODE). A key challenge is how to dis- cover intrinsic dynamics across multiple environ- ments while circumventing environment-speciffe mechanisms. Unlike prior work, we tackle more complex environments where changes extend be- yond function coefficients entirely different function forms. For example, we demonstrate the discovery of ideal pendulum's natural motion a² sin Og by observing pendulum dynamics in dif- ferent environments, such as the damped environ- ment a² sin(Ot) pWle and powered environment a² (0.) + p - Here, we formulate this prob- lem an invariant function learning task and pro- pose new method, known as Disentanglement of Invariant Functions (DIF), that is grounded in causal analysis. We propose a causal graph and design an encoder-decoder hypernetwork that explicitly disentangles invariant functions from environment--pecffie dynamics. The dis- covery of invariant functions is guaranteed by our information-bar principle that enforces the in- dependence between extracted invariant functions and environments. Quantitative comparisons with meta-learning and invariant learning baselines on three ODE systems demonstrate the effectiveness and efficiency of our method. Furthermore, sym- bolic regression explanation results highlight the ability of our framework to uncover intrinsic laws. Our code has been released as part of the AIRS
2018
et al. 2024) have focused on modeling dynamical systems by forecasting future states from past observations, often emphasizing rapid adaptation to new systems or improved model architectures. However, an important scenario in scientific discovery has been overlooked, i..,, which aims to identify shared motion patterns across dynamical systems observed in multiple environments. This task not only facil- itates scientific equation discovery but also holds potential for advancing the understanding and extraction of physical laws from observational data, such as images and videos, where physical laws are highly entangled. As the first step in invariant function learning, this paper focuses on ordinary differential equation (ODE) systems
The need for invariant function learning arises because data collected is often observed under varying environments and entangled with multiple factors. For instance, the oscilla- tion of a simple pendulum (Yin et al., 2021b) is commonly influenced by air friction; prey-predator system (Ahmad) 1993) can be affected by limited resources. These factors significantly hinder deep models from learning the true and invariant dynamics. Instead of capturing invariant dynam- ical patterns, deep models tend to be sensitive to trivial information and spurious correlations, leading to failures in identifying the true and isolated mechanisms. In light of this challenge, we explore an innovative setting called invariant function learning, which aims extract intrinsic mecha- nisms from data observed in multiple environments. Unlike prior work, we aim to tackle broader and more complex environments where changes extend beyond function coef- ficients to entirely different function forms. For example, we target the discovery of ideal pendulum's is natural motion a² sin Og by observing pendulum dynamics in different envi- ronments such as the damped environment a² sin(O:) - pwr and powered environment a² sin(A.) + P Wi shown in our motivation example T
library
1. Introduction
Deep neural networks (Goodfellow et al. 2016) have been widely used for predicting dynamical systems (Aussem
'Department of Computer Science& & Engineering, Texas A&M University, College Station, Texas, USA. Correspondence to: Shui- wang Ji <sji @tamu.edu>.
'This work introducee new perspective and focuses only on ODE systems. While the method has the potential to be extended to PDE systems, it is not directly applicable to them due to their continuous nature and the absence of multi-enviromment datasets designed by domain experts currently. Please refer to Appx. B for frequently asked questions.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Damped envirunment
Invariance
dhis
-oo sin (9)
49
day
dlf dhi Mki -0] sto
(NV)
(9.)
Oc
0c
W
Wi
We
?
Powered environment
d6
di
of (9))
dt (0))
d9
dal at
-0 stn (96)
2 we
Oz
O2
f
We
Wi
We
Wt
Figure I: Multi-environment Pendulum ODE systems. In this example, ODEs with different coefficients and function forms are used to extract their corresponding invariant functions (green).
Invariant function learning presents two key challenges. Firstly, invariant mechanisms are not isolated entities, and being intertwined with varying initial conditions, system parameters, and time makes them difficult to define or disen-
2.1. Ordinary Differential Equation Dynamical System
We describe a dynamical system using an ordinary differen- tial equation (ODF) as:
tangle. Secondly, existing invariant learning techniques (Ar- jovsky et al.
dx: dt
2019 Lu et al. Peters et al.
2021a la
Rosenfeld et al. 2020
f(x,),
(1)
201 which are primarily designed for cate-
gorical tasks, do not extend effortlessly to dynamical sys- tems, requiring the design of new invariant principles. To overcome these challenges, we formulate our invariant func- tion learning framework as causal graph where functions parameterized and isolated to be learned and disentan- gled. Furthermore, we propose an invariant function learn- ing principle and the implementation of an encoder-decoder hypernetoor (Ha et al. 2016) to identify the true invari- ant mechanism. Specifically, our contributions are listed as follows. (I) We introduce a new task, invariant function learning, aimed at scientific discovery. (2) We formulate its framework with causal foundations. (3) We propose in- variant function learning principle to identify true invariant mechanisms and design a method, Disentanglement of In- variant Function (DIF), with hypernetwork implementation. (4) To facilitate comprehensive benchmarking, we propose multi-environment ODE datasets and design several new baselines by adapting existing meta-learning and invariant learning techniques to our function learning framework.
where 2, X Rd includes hidden states of the system at time 1. / E F X TX is the derivative function of the dynamical system mapping the hidden states to their tangent space, where F is the function space containing functions that describe all dynamical systems with d hidden states.
Given proper time discretization, we consider T time steps denoted as / = lo, ,,,..,,,7........... The correspond- ing T-length trajectory can be written a matrix X RdxT Given the system hid- den states before a certain time step Te E N, denoted as X, X.,oT. . ing task aims to predict the future trajectory X.T.TT = [ "er,,...,,,,,111 It,_ random variables in boldface, egg., the matrix-valued ran- dom variable corresponding X is denoted as X; the func- tion is a realization of the function variable from a given function space. Full notations are detailed in Tab. 3
(atp Ftys. xty-1]
Ltz.,1l ..t. ,I E RRdXT. the forecast-
For theoretical analysis, we represent
2.2. Invariant Function Learning
2. Invariant Function Learning for Dynamical System
In this paper, we introduce a new task, invariant function learning (IFL). Specifically, given the prior distribution of trajectories, denoted as p(X), we consider a scenario where trajectories are observed under multiple environments. The trajectories observed in an environment E & are sampled from the conditional distribution p(X|e e = e). Given a tra- jectory X from environment e, our goal is to discover its
In this section, we first provide the background on ODEs, followed by the introduction and formulation of our invari- ant function learning task, along with the causal analyses that underpin our proposal.
2

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Table I: Comparison of environments with previous works in the pendulum ODE system. We list examples from 2 environments, the pendulum states and coefficients distribution within one environment, and inference time targets. The blue factors are changing across different environments. Full function environments are provided Appx.
Type
Environment é
Environment
Distribution
Inference
Coefficient environment
aff (9:) Pi fo (in pal
0b we) pl0, We. ex
-z sin(9:) psii
Function environment
G a² sin(6,)
6 sin 8: + pin)
[ = -a sin(
invariant function fer which generates the invariant trajec- tory X Je and X only include the shared mechanisms across all environments, thus capturing the underlying natu- ral laws unaffected by environmental factors. For instance, as illustrated in Fig. [] in the case of pendulum system with varying environmental effects such as frictions or power, the goal is to extract the natural motion of an ideal pendulum when excluding these external influences. A set of specific examples for the task is provided below, more examples are available in Appx. D
c
fe
X
Causal mupping Function prediction
X
Xo
fe
X
Function generation
Trajectory generation (Forecusting)
Function environments. The environments in this paper are different from those defined in CoDA (Kirchmeyer et al. 2022), LEAD (Yin al. 2021a). and MetaphysiCa (Mouli al. 2024). As shown in Tab. the enviromments/tasks used in previous works, namely, coefficient environments, are defined by the changes on the function coefficients and i.e., each environment contains only one function while different environments include functions with different co- efficients. In contrast, we consider more complex cases and define environments as the interventions on function forms, i.e., each environment can contain functions with the same function form and different coefficients, while differ- ent environments differ in function forms. Specifically, in Tab. [ while coefficient environment I consists of a single function f1, our function environment includes all the functions in form a (i.) pur where a ~ p(a)), p ~ p(p). Here, we model these functions as a function random variable fr.
Figure 2: Structural causal model. The causal data gener- ation process includes two phases: function generation and trajectory generation, which correspond to our two learn- ing phases in parentheses, namely, function prediction and forecasting. The gray nodes in the causal graph indicate observable variables.
2.2.1. CAUSALITY- BASEED DEFINITIONS
First challenge: causal formulation. In light of the first challenge, we aim to formulate the invariant function learn- ing and the dynamical system forecasting problem from a causal perspective. As shown in Fig. 2 we formulate the trajectory data generation process as Structural Causal Model (SCM) (Pearl 2009), where c, e, and Xn are ex- ogenous variables. All endogenous variables, except the function composition step fc, fc - f, are generated with extra random noises to model complex real-world scenarios. Please refer to Appx. C.T for more details. The optimization goal of the forecasting task is to estimate the true distri- bution p (X). As can be observed from the causal graph, our function learning framework can be considered two phases, namely, function prediction and forecasting. For the function prediction phase, similar to inverse problems (Lu) ct al. 2021b), given the observed trajectory Xp. the target is to reversely infer the invariant derivative function f that can represent the dynamics of the system, i.e., fitting pff Xp). Intuitively, taking Fig. ass an example, this phase aims at the reasoning of the function basis sin(..). -Wt, at and the coefficients a. p. After obtaining the derivative function f, the forecasting phase feeds it into numerical integrator with the initial condition X, for the X forecasting, which can be demonstrated as p(X X)). Note that the bold font variables are random variables instead of individual realizations.
Challenges. However, extracting such invariant dynam- ics presents two significant challenges. Firstly, invariant mechanisms are intertwined with varying initial conditions, system parameters, and time, making them particularly dif- ficult to isolate or define. Secondly, in dynamical systems, state values and their derivatives evolve over time, meaning that there no single invariant representation fixed across time steps. This requires defining invariant factors in function space, where functions can dynamical states. Conventional invariant learning techniques are not directly applicable in this context, as they are typically not designed to capture invariance in function spaces. These challenges necessitate the development of new function representations and the development of a novel invariant learning principle tailored to dynamical systems.
3

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Xo
MI.P Multilayer Perceptron
he...
h0a
TF
Transformer
Insert network parameters
MLP
Decoder
Forecast
TF
hoene RR
MLP
X
&e
2
he.w.
Hypernetwork
ML.P
Forecastor
96
Figure 3: DIF framework. è denotes outputs of the discriminator 9ø introduced in Sec. 3.2.1
Second challenge: function disentanglement. To handle the second challenge of extracting invariant mechanisms in dynamical systems, we aim to define invariant represen- tation in the function space, which requires the access to intermediate functions and the disentanglement formula- tions. With the two-phase function prediction to forecasting process, we explicitly expose the derivative function, al- lowing us to isolate the function prediction process and disentangle the learning of invariant functions. In the invari- ant function learning problem, we decompose the exposed function variable into an invariant function variable f. and an environment--peeciic function variable f.r as in Fig. [ These two functions caused by the exogenous factors and e, respectively. Intuitively, the variable includes the common and invariant mechanism, while e is the environ- ment variable determined by the observational environment of the trajectory, e.g., the pendulum dynamics can be ob- served different mediums (environments), such as air and water.
forecasting network, similar to traditional representation learning tasks, we aim to learn a function f E F RR + R For the function prediction, however, requires learning function that returns a function, E RdxT. F, i.e., learning hyper-function, which is enabled using hypemnetwrk.
3.1. Hypernetwork Design
Function prediction. To quantify the objective of the hyper- function, approximate its output function neural net- work with m parameters. The function space F consists of all possible neural networks with m parameters, and a func- tion ʃ E F can be represented as a vector in R". Thus this parameterization process introduces a hypernetwork struc- ture (Ha et al. 2016) into the implementation, as shown in Fig. 3 Note that since our parameterization transfers functions into the real number space, it is now possible to apply invariant learning techniques such IRM (Arjovsky) et al. 2019) and VREx (Krueger et al. 2021 where invari- ant (function) representations need to be extracted. In the following sections, we use F and omit R for simplicity.
Formally, the invariant function learning target is the dis- covery of f., which eliminates the effect of environments and obtains the invariant mechanism. However, from the d-separation perspective (Pearl 2009), since X, is the de- scendent the collider f fo and f.. given Xp. and fe are correlated/biased and not distinguishable, preventing the di- rect fitting f, given Xp. Therefore, we aim to identify this intermediate hidden variable by characterizing the unique properties of the prior distribution of p(f.). Theoretically, since f is the collider between f, and e, fc is expected to be independent of e. In addition, among all the functions that are independent of e, fo should be the most informative with respect the observed trajectories, which will be proved in Thm. [ This forms the foundation of invariant function learning.
Practically, since the number of parameters in a network is generally large, we propose to compress the invariant function representations into hidden representations, thus forming an encoder-decoder framework. Specifically, as shown in the Fig. 3 the trajectory encoder is transformer- based network with positional embedding design, denoted as hg... : Rdx'Ts + Z. Given the hidden representation from the encoder, we further encode an invariant function embedding & Z and an environment function embedding &e Z using two multilayer perceptrons (MLPs), denoted as ho and ha.... respectively. Then, aligning with our causal graph as Fig. 2 we combine & and &. by summing them as the function representation å E Z, which can be used for full dynamics prediction. Finally, we learn an decoder MI.P hoace to decode the function representation into m-dimensional neural network parameters, i.e., fc,
3. Disentanglement of Invariant Function
Following the two-phase function learning framework, we now propose the first method for IFL, Disentanglement of Invariant Function (DIF), with hypernetwrrk-baae imple- mentations of the two corresponding networks. For the
To facilitate theoretical analysis, we simplify the notations

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
and denote the function prediction process in the hypernet- work two functions f ho (xp) and fc ho, (Xp), where heg he. RxxT's + F; 0 = {Oene, Osne, Denw, Odee 0. {fene. Pine. }. In addition, we slightly abuse the notations of hig and ho, on random variables X, for simplic- ity, producing f - ho (Xp) and f. he. (Xp), respectively.
3.2.1. IMPLEMENTATION OF INVARIANT FUNCTION LEARNING PRINCIPLE
Following Thm. next we introduce the implementation and optimization process of our proposed networks. We first train the encoder and decoder of our hypernetwork by ap- proximating the trajectory distribution p(X), parameterized as p(X h (Xp) Xo). where we apply the cross- minimization. Given that our supervision signals only come from the ground- -ruth trajectories, we introduce a simple lemma for our optimization processes. The proof is pro- vided in Appx.
Forecasting. Given the produced neural network function f. we apply numerical integrator
forecastor, a function Jint that takes derivative function f and initial states Xo as inputs, to obtain X = gine (f.X)) (.))) + where is sam- pled from a Gaussian noise N (X; 0, o²1) introduced by calculation deviations. This forecasting formulation enables the following probability modeling, where we obtain the forecasting given realizations Xo and f as a Gaussian dis- tribution N X: gineenff.... xo),o=1) denoted p(Xlf, Xo).
Lemma 3.2 (ODE cross-entropy minimization). Givenfore casting model p p(x) it follows that the cross- entropy minimization between the data distribution p(X) and p(X (Xp,, Xo) is equivalen to minimizing mean square error ming Ex=plIX - X12 where X is sampled from p(X( (o((X)) Xo).
Therefore, in probability modeling, X is sampled from f( is worth noting that unlike in inference time and analyses, it is time-consuming to numerical integra- tors during training; therefore, we follow (Mouli et al. 2024) and train the model by fitting the derivative dX. = f(X,) de with numerical derivatives from the ground-truth X instead. For simplicity, we denote f()) as a neural network based derivative function with parameters f E R.M.
In order to discover invariant functions, we apply the invari- ant function learning principle, which requires maximizing the conditional mutual information between the predicted
function random variable f. = ho, (Xp) and f. Based on the derivation of Lemma 3.2 we have the following proposition.
Proposition 3.3 (ODE conditional mutual information max- imization). Given forecasting model p(X (Xp), Xo). it follows that the conditional mutual information maximiza- tion maxę, (Xp); is equivalent to minimizing mean square error mins, Ex~plIX - -X1 where X is the predicted trajectory sampled from p(X (ha (Xp). Xo)-
3.2. Discovery of Invariant Function
With the above hypernetwork design, we can now propose the discovery of the invariant function fe. Following the independence and information properties of f. discussed in Sec. 1.2.T we achieve invariant function learning with the following theorem, which our main theoretical result.
The proof is provided in Appx. C.4 Lemma 3.2 and Prop. esseentil transfers the mutual information maxi- mization of Thm. 3. into an implementable optimization of mean square error (MSE) loss, enabling the practical use of the invariant function learning principle. In addition, the independence constraint in Thm. 3.1 requires that the ex- tracted functions should not be over-informative or contain biased information from environments e. This indepen- dence constraint can be implemented in an adversarial way, where we require the environment prediction P(e(f.) to be as less informative as possible, i.e., minimizing the mutual information of / /ee; f.). Thus we introduce an environment discriminator 9a. a.k.a., (ef)ff), which aims to distinguish the environment of any function from F. The hypernetwork is trained adversarially to enforce f, as indistinguishable as possible. The theoretically analysis of this independence training is provided in Appx. C.5
Theorem 3.1 (Invariant function learning principle). Given the causal graph in Fig. D and the predicted function ran- dom variable f. = hag (Xp). it follows that the true in- variant function random variable fe can be from haz (Xp), where the optimal solution 0 is obtained through the following optimization:
0- arg max 1 (he. (X)): s.t. he. (Xp) TT e, (2)
0.
where ((::)) is mutual information that measures the in- formation overlap between the predicted invariant function random variable (X,) and the true jull-dynamics func- tion random variable f.
The proof in Appx. C.2 shows that the optimal solution is both necessary and sufficient to identify the true invariant function variable available Therefore, Thm. 3.1 establish guarantees and conditions for the function output f. of the hypernetwork to be the invariant function f., fulfilling the goal of the IFL task.
Objective. The overall optimization objective can be ob- tained with three training strategies. First, the training of the discriminator is conducted on both fo and fa to fully capture enviromment patterns. Second, we use the corre- sponding hidden representations of fo and fe. & and &g, as the input of the discriminator. Third, as we mentioned in
In the theorem, we consider any possible function h under our causal assumption instead of specific neural network realization.
5

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
See 3.1 during training, we fit derivatives instead of using an integrator.
systems, where hypernetworks are only employed as low- rank adaptors for new dynamical system trajectories, similar to the idea of LoRA (Hu et al. 2021). Our work differs from these in two key ways. First, we focus on discover- ing invariant functions rather than quickly adaptable ones or static scalar (Auzina et al. (023). Second, while meta- learning methods seek to learn a single meta-function, our framework learns multiple functions, represented by an in- variant function random variable fe. This distinction stems from our more complex environment definition, detailed in Sec. 2.2 From another aspect, learning an invariant function distribution instead of a single function can be considered as generalized meta-learning with an invariant function learn- ing goal.
dX.
f(x0)]
min E Ex-p
dt
+ d min Exxp)
Σ
dxx
fc(Xx))
6.
dt
(3)
+ Adis
min Ex-x log.o(
min 0.D.
-x log (o())
+ Ando max
Ex-p log (E))
where we denote 0. (Beney G fenui; 1; 0g- = {Benes Oino Please refer to Appx. E.2.2 for more details.
Current invariant learning methods (Arjovsky al. 2019
Efficient hypernetwork implementation. Last but not least, one the major challenges that limits the usage of hy- pernetworks is the implementation complexity. In this work, we propose a reference-based hypernetwork implementation to accelerate the running speed using only PyTorch (Paszke et al.] 2019) without re-implementing basic neural networks. The speedup compared to the naïve implementation and the vectorized functional implementation are 16.8x and 2x, respectively. Please refer to Appx. for implementation and experimental details.
et 2021a Rosenfeld et al. 2020 Krueger et al. 2021
Sagawa et al. 2019) follow the framework of invariant risk minimization (IRM) (Arjovsky et al. 2019), which was
inspired by invariant causal predictor (Peters et al. 2016). This invariant learning framework aims to learn a hidden in- variant representation or invariant causal mechanism (Pearl 2009) that generalizes across multiple environments, ensur- ing out-of-distribution performance. However, this approach cannot work on dynamical forecast tasks due to the lack of invariant function definition and the violation of the categor- ical data assumption (see Appx. B To address the issues, we introduce the causal assumption (see Fig. 2 that defines the invariant function space, and propose the corresponding invariant function learning principle and implementation.
4. Related Work
This work is inspired by the ideas and limitations of previous research in dynamical system forecasting, meta-learning, and invariant learning.
Symbolic regression methods (Brunton et al., 2016 Cran- mer 2023) and its extensions discover closed-form ODEs by fitting sparse combinations of basis functions. Recent symbolic discovery works add neural inductive biases for
Deep learning models are widely applied in many physi-
cal applications (Lusch et al., 2018 Yeo & Melnyk 2019
Kochkov et al. 2021 Chen et al. 2018|1 Becker al. 2023
graph-structured systems (Cranmer et al. 2020 Shi et al., 2022). Amortised approaches push scalability further, learn- ing to output symbolic expressions for hybrid systems in one forward pass (Liu et al.) 2024). Mechanistic neural networks adopt different framework to learn ODE repre- sentations (Pervez et al. 2024 Chen et al. 2025). However, all of these methods aim to find a separate symbolic equation for each individual system; in contrast, our goal is to learn components of the dynamies that remain invariant across systems and environments, complementary but distinct objective.
2024) including par- tial differential equations (PDEs) with the focus on the
d'Ascoli et al.
2023 Seifner et al.
multi-scale (Li et al. 2020 Stachenfeld et al. 202T), multi-
resolution (Kochkov et al.
2021: Wu et al. 2022).. and
2023) issues. Operator learning and neural operators (Gupta al. 2021
long-term stability (Li al. 2021 Lippe et al.
Kovachki al.
2023) popular for PDE estimations. Al-
though the ODE dynamical system does not contain the multi-scale problem that Fourier neural operator (Kovachki jet al. 2023) tried to solve, our framework can be considered as a kind of operator learning.
Meta-learning methods (Finn al. 2017 Rusu al. 2018 Li et al. 2017 Zintgraf al., 2019 Perez et al. 2018) aim to learn meta- pettapaarmeters that can be used across multiple tasks, where the meta parameters are generally learned to make rapid adaptations. In previous meta-learning stud- ies on dynamical systems (Kirchmeyer et al.] 2022 Wang ett al., 2022; Yin et al., (021a),, the objective was to find a meta meta-function that could quickly adapt to multiple new
5. Experiments
We conduct experiments to address the following research questions (RQs) and supplementary analyses (SAs). RQ1: Are existing meta-learning and invariant learning techniques effective for extracting invariant functions? RQ2: Can the proposed invariant function learning principle outperform baseline techniques? SA1: How do the full functions f and
6

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
the invariant functions f. differ in performance? SA2: Are the extracted invariant functions explainable and aligned with the true invariant mechanisms? SA3: How will perfor- mance change given different lengths of inputs and types of environments? (See Appx. F:2) SA4: Is the proposed hypernetwork implementation more efficient than previous implementations? (See Appx. G
5.3. Proposed Meta-learning and Invariant Learning Baselines
General dynamical system forecasting is different from in- variant function learning significantly, where they focus on how to adapt to new trajectories, which commonly requires further optimization, e.g., test-time adaptation (Mouli et al. 2024 or adaptations with meta information (Wang et al.
2022 Kirchmeyer et al., 2022). Unfortunately, under our scientific discovery setting, there is no extra information provided test time, making them inapplicable to this set- ting. Therefore, we construct 4 new baseline settings by transplanting the techniques of previous meta-learning and invariant learning to our proposed framework detailed in Appx. F.
5.1. Datasets
In our experiments, we introduce three multi entirmmenni (ME) datasets, ME-Pendulum, ME-Lotka- Volterra, and ME- SIREpidemic. These three datasets are generated by simula- tors modified from the DampedPendulum (Yin al. 2021b), Lotka- Voltera (Ahmad) I993), and SIREpidemic (Wang 2021). Specifically, each of the dataset's 's training sets includes four environments with 200 samples for each envi- ronment. Specifically, each environment corresponds to one specific environmental effect. ME-Pendulum contains three types of friction and one effect with external energy. ME- Lotka- Votka-Volterra modified the common predatory relationship into four modified relationships, e.g., adding resource limits. ME-SIREppidemic produces four conceptual epidemiology models with the same susceptible population to infected population relationship. In addition to the training set, we generate 200 samples with 50 samples for each environment as an in-distribution test set. Please refer to Appx. D for
et al.
We first adopt the meta-learning baseline MAML Finn et al. 2017), where we use its learned pera-parsssters for invariant learning evaluate whether the fastest adapted parameter is the invariant function parameter. Our second meta-learming baseline is CoDA (Kirchmeyer et al. 2022). where we replace its hypernetwork decoder with the full encoder-decoder hypernetwork in our framework to fit in our task. Aligning with the original CoDA paper, we set the dimension of the hidden representation to be 2. Similar to MAML, eliminate the adaptation part and use only the learned meta- meta- parameter for invariant state prediction.
more details.
For invariant learning baselines, we adopted the two most typical techniques, IRM (Ahuja et al., 2021) and VREx (Krueger et al.) (02T]. These two techniques are applied to the proposed framework, where IRM stands for the most typical definition of invariant learning, while VREx stands for the distributionally robust optimization baseline, which can be considered the generalization of Group- DRO (Sagawa et al. 2019)
5.2. Experimental Setup
To quantitatively evaluate the invariant function extraction performance, we need to remove the environment-related effects to generate invariant trajectories X as the invariant function ground-truth, e.g., we simulate new data by elimi- nating -pw; from -a² sin( - pwt in the ME-Pendulum dataset (Fig. [ To be more specific, a generated invariant trajectory X", aligning the causal graph, has the same sys- tem parameters as the corresponding biased trajectory X for the invariant part controlled by c, i.e., they have the same a in the pendulum example. This invariant trajectory genera- tion is being done the in-distribution test set that each trajectory X in this test set has its special corresponding invariant trajectory ground truth X
5.4. Quantitative Results
Similar to other scientific discovery tasks, such as drug discovery, constructing proper validation set is challenging Instead, with only observational data available, we generate invariant function candidates that can be further validated in real experimental settings, e.g., through the introduction of interventions (Pearl. 2009).
This test set design enables us to mimic the situation of scien- tific discoveries, where we only observe environment-biase data but are required to find and evaluate invariant function candidates. Specifically, given the biased Xp, the hyper- network is supposed to predict the corresponding invariant derivative function fc. Then, with a numeral integrator, the output of this invariant forecaster X- will be evaluated by comparing with the corresponding invariant trajectory ground truth X using normalized root mean square error (NRMSE).
To quantitatively compare these methods, we provide the corresponding hyper-parameter search spaces for each tech- nique in Appx. E and plot the results of random hyper- parameter sampling as distributions using Boxen plots. As shown in Fig. we compare the quality of the invariant function candidates based on their median, best result, and quantiles. Specifically, the median performance of our pro- posed method surpasses the middle candidates of all other approaches. The performance gaps are particularly notable on the ME-Pendulun and S-----II--iieiice datasets. For
7

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
1.75
1.9
NRMSE
1.30
635
Figure 4: Invariant trajectory prediction errors on 5 methods under 3 multi-environment ODE systems. For each method, provide model candidates with 80+ random syper-parameter selections in their searching spaces, i.e,, more than 1200 models in the figure.
VHART
0.39
14
64
GIDAL
9. 02
0:53
-11
NAML
-34
-839
-24
635
11
(a) X predictions using f
(b) X predictions using fa
Figure 5: Visualization of trajectory predictions on ME-Pendulum
example, on AI--eeuuu,mm, over 75% of our method's can- didates outperform the best results of MAML and CoDA, and more than 93.75% candidates of IRM and VREx. On ME-Lotka- Volterra, the median of our candidates still out- performs nearly all candidates from other methods. In ad- dition, as shown in the visualizations ME-Pendulum 5 our learned invariant function fe eliminates environmental resistances from the original trajectory (Fig. (a) and obtain a simple pendulum motion without attenuation (Fig. (5b). Both quantitative and visualization results demonstrate the superior capability of our method in extracting invariant functions (RQ2).
state ground truth X, comparing their results using the predicted invariant function fz and the full function f. As shown in Tab. 2 the performance on X using fe represents the core results of our invariant function learning approach. In contrast, the predictions on X using f serve baseline for the ablation study, where invari- ant function learning principle is applied. Additionally, the prediction eπror on X using fo implies the environmental information eliminated by the invariant function learning principle, while the NRMSEs on X using reflect stan- dard in in-distribution (ID) test errors. We observe that the ME-Lotka- MME-Loka-VVlterra dataset the most challenging, with NRMSE of 0.3881 in the ID test. This result is consis- tent with general deep learning outcomes in (Mouli et al.) 2024). given that ME-Lotka-Volee is more complex than its original version.
the
To address the first research question (RQ1), we observe that the invariant learning techniques, IRM and VREx, are generally more stable than the meta-learning baselines. Al- though IRM and VREx do not surpass MAML on ME- Lotka-Volterra Volterra, they outperform MAML on out of 3 datasets and are consistently better than CoDA. However, when compared to our proposed method, the best function candidates from these invariant learning techniques sub- optimal. This confirms that the general invariant learning principles fall short in the context of invariant function ex- traction, aligning with the discussions in Sec.
As expected, f performs well on X, while fo excels in pre- dicting X our ablation study, we compare the perfor- mance of invariant state predictions X across all datasets. The predicted invariant functions f. significantly outperform the full predicted functions f in terms of NRMSF, validating the effectiveness of the proposed invariant function learning principle. Furthermore, we conduct more strict ablation study with independent and f. training in Appx. F.1
5.5. Full Function v.s. Invariant Function
To analyze SA1, we benchmark our best invariant function learning models on the invariant state ground truth X and
8

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Table 2: Invariant function validation and symbolic regression. NAN denotes that the result is not applicable or not of interest.
Target
Function
MFF. PendPendulmm
MF.ILarka Volterra
ME-SIR Epiddemii
NRMSE
SR Explanation
NRMSE
SR Explanation
NRMSE
SR Explanation
dS -1.75
dp:
O 1.90
1.254pp
0.38gup
Fc
0.3561
de
of
0.0652
0.425,1,
dwt
97 sin
dgg 0.30g.
dRt, -0.008
d5; 0.2431. 1.2
dd
We
d2p -0. 7.76.
de
i
0.7884
dak 0,00
0.7919
dg d: 0.36
0.9867
0.40S;
dR. 0.000
d2
0.7994
NAN
0.6912
NAN
0.7641
NAN
i
0.1700
NAN
0,3881
NAN
0,0212
dSt
S.le
deg
dp op-
dt
51 + 11
f. GT
NAN
di day
NAN
d2
NAN
dt
sin (0.)
dgg
dt
6p.g. 79
dR:
8.
d2
5.6. Symbolic Regression Explanation
6. Limitations
Furthermore, to address SA2, we analyze the extracted in- variant functions f. by applying symbolic regression us- ing PySR (Cranmer 2023). As shown in Tab. 2 we com- pare the symbolic regression (SR) explanations of the ex-
While this work provides foundation for invariant func- tion learning in dynamical systems, several limitations and opportunities for future exploration remain. These include extending the framework to more complex entanglements, exploring applications in PDE systems and developing com- prehensive benchmarks. Additionally, broader applications such as generalizable physics learning and foundational model development represent exciting directions for further research. Please refer to Appx. B for more discussions.
tracted invariant functions f. with the true invariant func- tions f.. On the ME-Pendulum dataset, the frictionless pen- dulum function is nearly perfectly extracted, with 0.99w: matching wi and -0.97a² sin (0,) closely approximating the true 2 sin (..). Given the complexity of the ME- Lotka-Volterse dataset, the extracted invariant functions fc are non-trivial and significantly outperform the full fune- tion f. On the ME-SIREppieemie dataset, the near-perfect NRMSE for the invariant state indicates that the invariant function must have been correctly extracted. However, al- though the expression for dR, is correct, the extracted ex- dt
7. Conclusion
In this work, we target addressing the challenge of the in- variant mechanism discovery ODE dynamical systems by extending invariant learning into function spaces. We intro- duce
task, invariant function learning, which aims to extract the invariant dynamics across all environments with different environment-<pecti function forms. We design causal analysis based disentanglement framework DIF to expose the underlying invariant functions. Additionally, we propose an invariant function learning principle with the- oretical guarantees to optimize the framework and ensure effective invariant function discovery. Our experiments, in- cluding invariant trajectory validations, visualizations, abla- tion studies, and symbolic regression analyses, demonstrate the effectiveness of our method. Finally, as discussed in Sec. [ the introduced invariant function learning task has wide application scenarios and many challenges remain to be addressed. We expect that our work will shed light on numerous future explorations in this field.
pressions for dS. and dl, do not precisely match the dr pected f.. Specifically, the coefficient from dr does not equal the inverse of the coefficient from बi though this dis- crepancy should be constant. These mismatches attribute to the limitations of PySR given the large number of variables and samples Future work could explore incorporating stronger inductive biases, similar to physics-informed ma- chine learning (PIML) methods (Mouli al., 2024] Yin et al. 2021b, Cranmer et al.) 2020), to address these chal- lenges. Please refer to Appx. F3 for symbolic comparisons
di
with all baseline.
'Superior PySR explanations indicate great invariant function learning results, but effective invariant function learning results might not lead to good PySR explanations.

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Acknowledgments
Cho, K. Learning phrase representations using mn encoder- decoder for statistical machine translation. arXiv preprint arXiv:14066 1406.0778, 2014.
This work was supported part by National Science Foun- dation under grant CNS-2328399 and ARPA-F under grant IAYIAX000053.
Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01 00000.0112.... 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cranmer, M., Sanchez Gonzalez. A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., and Ho, S. Discovering sym- bolic models from deep learning with inductive biases. Advances neural information processing systems, 33: 17429-17442, 2020.
References
d' Ascoli, S., Becker, S., Mathis, A., Schwaller, P., and Kilbertus, N. Odeformer: Symbolic regression of dy- namical systems with transformers. arXiv preprint arXiv:2310.0557,3 2023.
Ahmad, S. On the nonautonomous volterra-lotka competi- tion equations. Proceedings of the American Mathemati- cal Society, 117(1):199- 1944 1993.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional conference on machine learning, pp. 1126-1135. PMLR, 2017.
Ahuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J.-C., Bengio, Y., Mitliagkas, I., and Rish, I. Invariance princi- ple meets information bottleneck for out-of-distribution generalization. Advances in Neural Information Process- ing Systems, 34:3438-344 -3480, 2021.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 17(59): 2016.
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez- Paz, D. Invariant risk minimization. arXiv preprint 1.07:0990.0282) 2019.
Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. In International conference machine learning, pp. 1243- 1252. PMLR, 2017.
Aussem, A. Dynamical recurrent neural networks towards prediction and modeling of dynamical systems. Neuro- computing, 281133) 1999.
Auzina, A., Yildız, ç., Magliacane, S., Bethge, M., and Gavves, E. Modulated neural odes. Advances in Neural Information Processing Systems, 36:44572 44594, 2023.
Giannakis, D. Data-driver spectral decomposition and fore- casting of ergodic dynamical systems. Applied and Com- putational Harmonic Analysis, 47(2):338-399 2019.
Becker, S., Klein, M., Neitz, A., Parascandolo, G., and Kil- bertus, Predicting ordinary differential equations with transformers. In International Conference on Machine PMLR, 2023.
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Deep learning, volume 1. MIT Press, 2016.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(1 139-144,, 2020.
Learning, pp.
Brunton, S. L., Proctor, J. L., and Kutz, J. N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15)3392-335 ,937,293
Gupta, G., Xiao, X., and Bogdan, P. Multiwaveler-bseed operator learning for differential equations. Advances in neural information processing systems, 34:24048-2406 2021.
Chen, J., Yao, D., Pervez, A., Alistarh, D., and Locatello, F. Scalable mechanistic neural networks. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URL https://openeeiie..efffommm Oazgf8A24z
Ha, D., Dai, A., and Le, Q. V. Hypernetworks. arXiv preprint arXiv: 09199,999,...... 2016.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
10

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Kirchmeyer, M., Yin, Y., Donå, J., Baskiotis, N., Rako- tomamonjy, A., and Gallinari, P. Generalizing new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pp. 11283-11301 PMLR, 2022.
Lusch, B., Kutz, N., and Brunton, S. L. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950, 2018.
Mouli, S. C., Alam, M., and Ribeiro, B. Metaphysica: Improving ood robustness in physics-informed machine learning. In The Twelfth International Conference on Learning Representations, 2024.
Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner, M. P., and Hoyer, S. Machine baaning-acceleratee com- putational fluid dynamics. Proceedings of the National Academy of Sciences, L18(1)):221018841188 2021.
Ortiz, J. J. G., Guttag, J., and Dalca, A. Non- proportional parametrizations for stable hypernetwork learning, (XXiv2334.007655 2023.
Kovachki, N., Li, Z., Liu, B., Arizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 24(89):1-97, 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Bi- nas, J., Zhang, D., Le Priol, R., and Courville, A. Out-of- distribution gencralization via risk extrapolation (REx).
Pearl, J. Causality. Cambridge university press, 2009.
In International Conference on Machine Learning, pp- 5815-5826 PMLR, 2021.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general con- ditioning layer. In Proceedings of the AAA/ conference on artificial intelligence, volume 32, 2018.
Li, Z., Zhou, F., Chen, F., and Li, H. Meta-sgd: Learn- ing to learn quickly for few shot learning. CoRR, abs/1707.09835,, 2017. URL http://anxiv.org/ab//1777 09835
Pervez, A., Locatello, F., and Gavves, E. Mechanistic neural networks for scientific machine learning. In Proceedings of the 41st International Conference on Machine Learn- ing. pp. 40484 40501, 2024.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equa- tions. arXiv preprint (tt:::000008999) 2020.
Peters, J., Bühlmann, P., and Meinshausen, N. Causal in- ference by using invariant prediction: identification and confidence intervals. Journal of Royal Statistical Soci- ety: Series (Statistical Methodology), 78(5):947-1012, 2016.
Li, Z., Liu-Schiaffin,, M., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandku- mar, A. Learning dissipative dynamics in chaotic systems. arXiv preprint arXiv: 106.88806.06888, 2021.
Rosenfeld, E., Ravikumar, P. and Risteski, A. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761 2020.
Lippe, P., Veeling, B. S., Perdikaris, P., Turner, R. E., and Brandstetter, J. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. arXiv preprint arXiv:2308.05732 2023.
Rusu, A. A., Rao, D., Sygnowski, J., Vinyals, O., Pas- canu, R., Osindero, S., and Hadsell, R. Meta-learning with latent embedding optimization. arXiv preprint 1eoe:8807.0900 2018
Liu, Y., Magliacane, S., Kofinas, M., and Gavves, S. Amor- tized equation discovery in hybrid dynamical systems. In International Conference Machine Learning, pp. 3 31645-31668, PMLR, 2024.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gener- alization. arXiv preprint /9/ 1,08731, 2019.
Lu, C., Wu, Y., Hernández-Lood Lobato, J. M., and Schölkopf, B. Invariant causal representation learning for out-of- distribution generalization. In International Conference Learning Representations, 20
Seifner, P., Cvejoski. K.. Körner, A., and Sánchez, R. J. Foundational inference models for dynamical systems, 2024. URL http:/pp/aarii.orraaa////.....5
Lu, L., Pestourie, R., Yao, W., Wang, Z., Verdugo, F., and Johnson, S. G. Physics-informee neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6)BB1 105-B11111 1---B1133 132, 2021b.
Shi, H., Ding, J., Cao, Y., Liu, L., Li, Y., et al. Learning sym- bolic models for graph-structured physical mechanism. The Eleventh International Conference on Learning Representations, 2022.

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Singh, S., James, M., and Rudary, M. Predictive state representations: A new theory for modeling dynamical systems, arXiv preprint arXi 4167, 2012.
Stachenfeld, K., Fielding, D. B., Kochkov, D., Cranmer, M., Pfaff, T., Godwin, J., Cui, C., Ho, S., Battaglia, P., and Sanchez-Gonzalez, A. Learned coarse models for effi- cient turbulence simulation. In International Conference on Learning Representations, 2021.
Sudhakaran, S. S. hyper-nn. https:/gginub.com//hhaamsss) hyper-nn, 2022.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
von Oswald, J., Henning, C.,, Grewe, B. F., and Sacramento, J. Continual learning with hypernetworks. In Interna- tional Conference on Learning Representations, 2020.
URL hutps://xxiv.orraab///1.....
Wang. R., Maddix, D., Faloutsos, C., Wang, Y., and Yu, R. Bridging physics- and data-driven modeling for learning dynamical systems. In Learning for dynamics and control, pp. 385-398. PMLR, 2021.
Wang, R., Walters, R., and Yu, R. Meta-learning dynamics forecasting using task inference. Advances Neural Information Processing Systems, 35:21640-21655 2022.
Wang, W.-X., Lai, Y.-.., and Grebogi, C. Data based identi- fication and prediction of nonlinear and complex dynami- systems. Physics Reports, 644:1-76, 2016.
Wu, T., Maruyama, T., Zhao, Q., Wetzstein, G., and Leskovec, Learning controllable adaptive simula- tion for multi-scale physics. In NeurIPS 2022 Al for Science: Progress and Promises, 2022. URL https: hompeneviee..e/fooumni..PPhhheee
Yeo, K. and Melnyk, I. driven simulation of noisy dynamical system. Journal of Computational Physics, 1776::212-122 1231, 2019.
Deep learning algorithm for data-
Yin, Y., Ayed, I., de Bézenac, E., Baskiotis, N., and Galli- nari, P. Leads: Learning dynamical systems that general- ize across environments, Advances in Neural Information Processing Systems, 34:7561-7573 2021a.
Yin, Y. Le Guen, V., Dona, J., de Bézenac, E., Ayed, I., Thome, N., and Gallinari, P. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 1021(12):22011,, 2021b.
Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and White- son, Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp. 7693- 7702. PMLR, 2019.
12

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Appendix of Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Contents
Notations
14
FAQ & Discussions
15
Invariant Function Learning foundation
18
Structural Causal Model
18
Proof of Invariant Function Learning Principle
18
Proof of ODE Cross-entropy Minimization
19
Proof of ODE Conditional Mutual Information Maximization
20
Theoretical Justification for Adversarial Training
Datasets
23
D.1 Basic Sctup
23
D.2 ME-Pendulum
23
D.3 ME-Lotka Volterra
23
D.4 ME-SIREpidemic
23
Experimental Details
25
Baselines
25
Baseline details
26
E.2
Disentanglement of Invariant Function Sctup
27
E.2.1
Architecture
27
F.2.2
Training Objectives
27
E.2.3
Metric
28
Software and Hardware
28
Supplementary Experiments
28
F11
Ablation Study
28
Input Length and Environment Analysis
29
Symbolic Regression Explanation Comparisons
30
Extra Visualizations
30
Efficient Hypernetwork Implementation)
30
G.1
Efficiency Comparisons
31
13

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
A. Notations
For the ease of reading, we providing a table including major notations below for reference.
Table 3: Notation table
Notation
Explanation
24 x TX R d
Dynamical system states at time
Dynamical system state space
Tangent state space
Real number space
The number of state Time
Te
The first future time step A trajectory; A state matrix with T-step states Past states before time step Te
X
X,
xe
invariant trajectory
R. X
The predicted trajectory of X, Xp. and X (by a model)
X, Xp X
The matrix-valued random variable of X, Xp. and X
X, Xp, X
The predicted matrix-valued random variable of X, Xp, and X
derivative function (of an underlying dynamical system)
fo So f Se L, Po
An invariant derivative function
An environment derivative function
The predicted functions of f. fc. and fe (by model)
The derivative function random variable of f. for and The predicted derivative function random variable of f. Sc- and fe
F 2/2c/2e
Function space/Functional vector space
A predicted full/invariantenniiroment hidden function rep- resentation
2
Hidden function space/Hidden functional vector space
A hypemetwork
H
Hypernetwork function space
p())
probability distribution over random variable
1(5.)
Mutual information between random variables
H(X)
Shannon entropy of the matrix- valued random variable X
Ex-r[f(X)] or Ef(X)
Expectation of fXX) with respect to pX)
X~ P
Random variable X has distribution P
X~ X~p(X)
X sampled from distribution (X)
Exxr[f(X)] or Ex Expectation of f(X) with respect to p(X) assignment/A substitution rule
())
(a
symbol without an assigned value
aa
A symbol with an assigned value
14

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
B. FAQ& & Discussions
To facilitate the reviewing process, we summarize the answers to the questions that arose during the discussion an earlier version of this paper.
The major updates of this version are reorganized theoretical studies, causal graph details, experimental analyses. We include more related field comparisons to distinguish different settings. We also cover the position of this paper in literature and the main claims of this paper. Finally, we will frankly acknowledge the limitations of this paper, explain and justify the scope of coverage, and provide possible future directions.
Q1: Can this method be applied to PDE systems?
A: While this method has the potential to be extended PDE systems, three critical challenges currently prevent its direct application:
Lack of multi-environment PDE datasets: Unlike domain adaptation and generalization tasks, multi-envirrnment datasets for PDE systems are not yet available. Although we constructed multi-ennviooment datasets for ODE systems, extending this to PDEs is significantly more complex and requires domain-specific expertise. Unfortunately, designing such datasets is beyond the scope of this paper.
2. PDE-specific challenges: Due to the continuous nature of PDEs, applying invariant function learning PDE systems requires addressing multi-scale and multi-resolution problems. Scaling up to PDEs also introduces different training dynamics, which may necessitate additional techniques to stabilize and accelerate training.
3. Interpretability challenges: As demonstrated in Appendix [3 we employ symbolic regression to provide conceptually understandable explanations for sanity checks. However, this approach does not extend naturally to PDE systems, which would require the development of new interpretability methods tailored invariant function learning in PDEs.
Q2: Is f. one deterministic true function? What is the difference between X and X; f and f?
A:
1. Is f. deterministic function? No. fg is a random variable sampled from the structural causal model (SCM) (see Fig. 6). obeying the Markov property of graphical causal models. Consequently, all information-theereei analyses in this work are non-trivial.
2. Difference between X and X; f and f. X RdxT represents single realization sampled from the matrix-shaped random variable X, i.e., one trajectory. Similarly, f is realization of the random function f.
3. Other notation-related questions. Our notation follows ICLR standards. Please refer to our notation table (Table It is crucial distinguish between random variables and their realizations.
Q3: What are the differences between coefficient environments and function environments?
A: The primary difference lies in which factors vary across multiple environments.
1. Example: Consider the pendulum motion as example. A coefficient environment includes only single function. Any change in the rope length or friction coefficient defines a new environment. In contrast, function environment encompasses all functions that share the same functional form but differ in parameters such as rope length and friction coefficient. As long as these pendulums experience the same type of friction, they belong to the same function environment.
2. Notation: Since a coefficient environment contains only one function, we use f to represent that function. Conversely, a function environment consists of multiple functions. Theoretically, the number of functions within a function environment is infinite. Thus, we use the random variable f to capture the function distribution within a function environment.
15

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
It is important to emphasize that single function does correspond to single trajectory. Even with the same Ttstep discretization, function can generate infinitely many trajectories depending different initial conditions.
Q4: Why don't we fit trajectories with symbolic regressions and observe the invariant function?
A:
Regarding alternative strategies like find symbolic expression different environments and then obtaining the same part fo through observing. the most difficult part observing. The reason is direct: the fitting of f tends to capture both invariant and esvironment-specific aspects, which leads spurious correlations, so that the final equation forms from different environments look significantly different. That is, after extracting equations from different environments, it is challenging to find the common part directly since the common parts are blended with environment parts and look like not disentangable.
Our DIF method explicitly enforces the separation of invariant dynamics from environment-specifc factors, providing a more reliable basis for scientific discovery. This advantage become especially important in scenarios with complex environment effects or when specific invariant mechanisms need be identified.
Q5: Why traditional invariant learning cannot be applied directly?
A:
Current invariant learning methods (Arjovsky et al. 2019 Lu et al. 2021a Rosenfeld et al. 2020 Krueger et al. 2021
Sagawa et al. 2019) follow the framework of invariant risk minimization (IRM) (Arjovsky et al. 2019). which was inspired by invariant causal predictor (Peters et al., 2016). This invariant learning framework aims to learn a hidden invariant representation or invariant causal mechanism (Pearl) (2009) that generalizes across multiple environments, ensuring out- of-distribution performance. However, this approach cannot work on dynamical forecast tasks due the lack of invariant function definition and the violation of the categorical data assumption. To be more specific, first, invariant functions cannot be naturally defined in the real number vector space. Second, invariant learning commonly assumes the prediction results are categorical, where a single invariant representation can fully determine the corresponding label. However, this assumption is violated in dynamical system forecasting, where the invariant mechanism is only partially responsible for the output. In this case, the IRM principle can not hold even when the invariant function ground truth is provided. To address the issues, we introduce the causal assumption (see Fig. 2 that defines the invariant function space, and propose the corresponding invariant function learning principle and implementation.
Q6: What is the position of this paper? What can be covered by this paper?
A:
Position: This paper introduces the concept of invariant function learning, motivated by the function learning requirements in physical systems.
Scope of this paper: multiple perspectives.
The primary focus of this work is establish invariant function learning in ODE systems from
1. Model design: We propose the first invariant function learning method.
2. Theoretical analysis: This paper builds upon the well-known invariant learning principle. We establish the foundation of invariant function learning, propose ODE-based invariant learning structural causal model (SCM) with minimal assumptions, and provide theoretical guarantees for invariant function discovery.
3. Function environments and datasets: We introduce the concept of function environments. To systematically evaluate invariant function learning, we construct multiple multi-environment ODE systems.
4. Adapted baselines: We detail the design and implementation of adapted baselines from invariant learning and meta-learning frameworks.
5. Empirical analysis: interpretability assessments, and multi pellppipereeettvee ablation studies on ODE systems (see Appx.
We conduct extensive empirical studies, including quantitative comparisons, visual analyses,
Q7: What cannot be covered by this paper? What are the limitations of this paper?
16

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
A: This paper aims establish solid foundation for invariant function learning from multiple perspectives, as described in Q4. However, exploring invariant function learning is complex and expansive task that cannot be fully addressed in single work. Below, we outline the key limitations of this paper to guide future research directions.
1. Invariant function learning in PDE systems: This work focuses solely on invariant function learning in ODE systems. Extending this approach to PDE systems remains an open challenge. The reasons and current obstacles in this setting have been elaborated in FAQ Q1.
2. High dimensional system with graph structures: One interesting direction considering graph-like interactions has been partially explored by Shi et al. (2022); Cranmer et al. (2020). These systems are generally nosier and prone to affected by environment effects. Therefore, extending our methods to this topic is meaningful future direction.
3. Broader range of applications: Unlike meta-parameterss the learned invariant functions exhibit broader adaptabil- ity. For instance, a discovered physical law can generalize across various systems. Future research could explore applications of invariant function learning, such as extracting generalizable physics laws from vidcos or designing physics-aware aware agents. Additionally, would be interesting to investigate whether invariant function learning can contribute to the development of foundational models in physics.
17

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
c
f
X
Causal mapping Function prediction
f
X
Xo
fe
X
Trajectory generation (Forecasting)
Function generation
Figure 6: Structural causal model.
C. Invariant Function Learning foundation
C.1. Structural Causal Model
In this section, we discuss the trajectory generation process under the Structural Causal Model (SCM) assumption in Fig. To begin with, this SCM is directed acyclic graph (DAG) with the following components:
Exogenous variables U = {c, e, Xo, co: Ees Ep. E} are not caused by any variables within the model and are from their own independent distributions. Here co: Ses Eps are noise terms introduced during the function generation and trajectory integral.
Endogenous variables V = X,,
X) are caused by the causal mappings within the model.
Structural equations define the direct causation in the model.
fc = ge(c, Ee)
ge(c,EE) Ee)
- f = Icomp (fo, fe)
X Fint (for Xo)
- X, = Iint (f, Xo) +
X gint (f, Xx) +e
where gint is a T step integrator, while Th only applies integral for Tg steps. While Jc, 9c, 9comp are assumed to be unknown, fint is assumed to be an ideal integrator with an extra random noise to model the real world situations. Note that the cause of X can be written as f. and f, without side effects, so used for analytical purposes. Therefore, Jeomp is a conceptual function composition without introducing noises.
C.2. Proof of Invariant Function Learning Principle
Theorem C.1 (Invariant Function Learning Principle 3.1.
Given the causal graph in Fig. 2and the predicted function random variable f. = ha, (Xp), the true invariant function random variable is f. hog (Xp). where 0% is the solution to the following optimization problem:
0- arg max I [ha, (X)); subject to ba_ (Xp) 11 e,
(4)
0.
where .(;))) denotes mutual information, which quantifies the informational overlap between the predicted invariant function hgg (Xp) and the true full-dynamics function
Proof. Existence:
We first prove the existence of a solution 8: to the optimization problem, such that f. hgg (X,). To establish this, we proceed by contradiction. Assume no such 0 exists, implying that T(fe; f(X,) is not maximized. Then, there must
18

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
exist some f. such that fo Il and [(;;f [(fe; ((o): Given the mutual information expression [(f;; (())
(f (ffffe, Xo), this inequality implies:
H(f..) > (()))))
(5)
Since Xo is independent of f and fo (as per Fig. 2. and given that f = ge fe) implies H fe) = 0, we derive:
H(TE.) (f.)).
(6)
Similarly, for f.+ we have:
H(A()) 2 ((..
(7)
Combining these results with Eq.
5 we obtain:
H(f.) > H(f.lf)).
(8)
which contradicts the independence condition f- IL e, as this would require (f.) = (fel(.). Therefore, a solution 0
exists, satisfying fc = hog (Xp)-
Uniqueness: We now prove that for any solution 0% of the optimization process, it holds that f. hag (Xp).
We a proof by contradiction. Assume that there exists another solution f' A that satisfies the independence constraint and achieves the maximum mutual information. By assumption, we have (c) (elf() and [(f;; (())) f[x,) which implies (f]..) (I).)).
Since f = (fc, fe), we expand the entropy terms:
((fff.) (ff,ff.lf.. H(f.).
(9)
and
H(f(() = H(fe,falt) H(f.(()) + ((f.lf.) = H(f.(()) + H(f.).
(10)
Substituting (fff.)) (ff(')) into these equations, find:
H(f.(()
(11)
Based on this, we now aim to prove that f. = f.. First, given (f.lfe) 2 O, we need to show that ((f.lf)) = 0. Assume that H (f. > 0. this case, f. can determine f. while containing more information than f., all while remaining independent of f.. This would imply that H (f'(f) > 0, where = gf((", f.). which would turn affect the corresponding prediction, leading to (X{ >
Such situation would violate the MSE minimization condition.
Therefore, must have both H(f.ffe) 0 and (f. 0. This implies that fo and are isomorphic functions, i.e., there exists bijective function db such that f. = ga (f.).
Furthermore, since minimizing the MSE of X is equivalent to minimizing the MSE of J.X. de given Xo, this minimization ensures that, for the same fe, fx( = (X) for all X. As result, the bijective function 9b must be the identity mapping, and we conclude that f. = f. the support of p(X).
Thus, for any solution 0 of the optimization process, it follows that f. ha: (Xp).
C.3. Proof of ODE Cross-entropy Minimization
Lemma C.2 (ODE cross-entropy minimization 3.2 Given our our forecasting model p(X(ha((X)) Xo), follow that the cross-entropy minimization between the data distribution p(X) and p(X ho (Xp))) (Xp),X)) X.) is equivalent minimizing mean square error ming Ex=pIIX X 112, where X is sampled from p( (h((X))) (Xp), Xo)-
19

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Proof. The forecasting optimization goal is use framework approximate the data distribution pl pXX,, param- eterized p(X(ha X p), Xn), where we apply the cross-entropy minimization, i.., (e..H((()) p( (Xp,) Xo)) -Ex~p [log p X hoa (Xp), p), Xo)]. Furthermore, this negative log-likelihood optimization can be further reduced to the
common mean squared error (MSE).
min Ex-nBx-plla 112 0
(12)
where X
p(X(ho(X) Xo). Since the distribution p(X (Xp): [.p):oo) Xo) is modeled as a Gaussian
N (X,,), Xo), 1) (Sec. 3.1 have X 1 e, where N (0, o0,
min Ex Ex flog p x (Xp), Xoo]
min x
1 log (ogg2 2)
2
+
20 || ml12
(13)
n log log((2oT*)) (2TO²)
min Ex-p
X
+ 2
20
Since log(2TT²?) is constant, ignore it in the minimization process.
min Ex-p
2x X - 1112
"
(gg(2rTo))
2
min Ex-p
20 | - (X -
min Ex-p
2 XX- - X + +ell
(14)
min Ex-p
20 (| 212 + 11e112 + 2(X - x)
min Ex-p
X - {11
2 R.
-
Ex-p(X E.
20
1 Ex>pllle is a constant; Exup Here, 2n2
1 (X - -X)"' = 1 Exxp(X - E.(e 0 since e is independently
sampled with zero mean. Therefore,
min Ex~p -p
20 X - 811
+
20 E.lle
jtt
1 Ex - -8)
min Ex~p
20 ||X - -1112
(15)
min Ex~p
|X - {11
This reduction connects the information theory and the practical MSE optimization, which further helps us to transform the mutual information maximization into similar MSE optimization below.
C.4. Proof of ODE Conditional Mutual Information Maximization
Given forecasting model pho (Xp), X)), it follows that the conditional mutual information maximization max., I(ho, (Xp); is equiv-
Proposition C.3 (Proof of ODE conditional mutual information maximization
alent to minimizing mean square error ming. Ex=p - & where X= is the predicted trajectory sampled from (X,), (X)).o)).
Proof. According to Lemma
3.2
we
can reduce a negative log- ligeliieddoodd
minimization
ming. -x log lo (X (Xho, ho. (Xp), Xo) to ming, E Ex-pll X
20

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Therefore, only need prove the equivalence between the -Bx-p log p( (app(((hpp) (Mp) Xo) minimization and the T(ho, (Xp); f(xo) maximization. It follows that
max T(ho, (Xp);f(o)
(16)
max H() - H(flho, (Xp), Xo). 6.
Since (f) is constant, we ignore it in the maximization process and obtain
max (f (f]he (x)))X0)
0.
max 11( (Xp):
(17)
0.
min Hffhhg (Xp) Xo)-
0e
Since X = (f, Xo) + where e is an indepedent random noise, H ha, (x,), = (e = (e). Since given specific 8. (ha. (X,) 0, with the conditional entropy chain rule, follows that
min (f]hha (X Xx)
min Hff, X Xha, (X xp),Xo Xo) HX(haa, (Xp), Xo.
uin [f,, //a x,),Xo) ) -+(()
6.
min (f,, X Xho Xo)
6.
min Hff)ho (Xp), Xo, X) H(X (X,), Xo)
(18)
8:
min (f X) H( (Xp), Xo)
min H No. xx),
6-
min Ex x X log (X hgg (Xp)) Xo)
6.
min Ex-p log p (x hog (x,
where X,, X) is a constant. Here, we finish building the equivalence between the function conditional mutual information maximization and the trajectory negative likeliihddo minimization. Using the proof Lemma 3.2 our final optimization goal can be reduced to
min Ex X 11.
(19)
C.5. Theoretical Justification for Adversarial Training
To incorporate the independence constraint, enforce the condition fc 11 where f. ho, (Xp) random variable, not a realization. Since f. is equivalent to (e;; f.) O. and T(e; fz.) 2 0, the objective [(e; f.) reaches its minimum when and only when f. IL e. This leads define minimizing / (e; fo) our training criterion.
) is the predicted function
Definition C.4. The environment independence training criterion is
0- = arg min fc;
(20)
0.
This training criterion can be used directly, since the mutual information I(e; f.) E log P(effx) P(e) while P(e f.) is unknown.
Following Proposition 1 in GAN (Goodfellow et al. 2020), we introduce an optimal discriminator Iø F + E with parameters & to approximate the unknown P(e)f.) as Pa (c)f.), minimizing the negative log-likelihood -IE log (e(((.))
We then have the following two propositions:
21

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Proposition C.S. For Oe fixed, the optimal discriminator
arg min -E log Pale(f.))
(21)
This proposition can be proved straightforwardly by applying the cross-entropy training criterion.
Proposition C.6. Denoting Kl-ddivergence as KL.[-I-). for 0 fixed, the optimal discriminator & is o¹, such that
KL P( = -.
(22)
Proof. Given fixed 0.. both [(c;f,) and (e) are constants, Therefore, we have:
arg min -E log Palelf.)
arg min f(e; - E [lo Polefta) H(e)
(23)
arg min KL, p(efft)lPP((eff)]
Thus, minimizing the negative log-likelihood of Po (elfcc is equivalent to minimizing the KL divergence between P(effe) and its approximation (e(f.). Since KL divergence is bounded by O, we have KL ((elfe))lPP This concludes the proof.
With these propositions, the mutual information be computed with the help of the optimal discriminator o², According to Proposition C.6 have:
I(e; fe) = E log Pa- (elfa)
+ H(e) + KI. P(eff.))lPa-(e(( (elfa))
(24)
E log Po. (c(fo)
+ (e) +
Thus, by disregarding the constant H (c). the training criterion becomes:
0 arg min [(e;f.)
0.
arg min E log Po. (elf.)
(25)
9.
max E log (oo())((()))
arg min
0.
where Po(fff) is the probability modeling of Jr. Therefore, the log- likelihoddood adversarial training can enforce the independence he, (Xp) IL
22

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
D. Datasets
D.1. Basic Setup
We conduct experiments on the proposed three mulli-environment datasets -eE-Pueuu,u, ME-Lotka-Volterra, and ME- SIREpidemic. Each of these datasets includes 1000 samples, where 800 and 200 samples are assigned to training set and test set, respectively. Each training set has 4 environments where 200 samples are generated in each environment. Each sample is observed over 10 units of time, and each time is discretized by regularly-spaced discrete time steps from ty to Ir, where = 99, i.e., there are 100 time intervals of 0.1 unit of time each. A sample generation process is controlled by set of ODEs with common parameters W ~ UdW low W liggh) and environment-speccnn speccffc parameters W ~ U(W low W high)) sampled from their uniform distributions, e.g., in the pendulum system, W = {o}} and W = (p), where or ~ l U(ooow (thigh) and p ~ (Ptows Phigh). Note that each environment has its specific function form with its environment-specifie parameters W
The environment split is the same in the test set. The only difference is that in the test set each sample X has one additional prediction target X with only the invariant dynamics. For example, the invariant trajectory X of the one generated by -a² sin O2- pwa will be created by -a² sin Oz.
D.2. ME-Pendulum
ME-Pendulum is motivated by the DampedPendulum system Yin et al. and angular velocity of the pendulum at time i, where we have W invariant ODE is dea dt wh, di aa sin (0t). As shown in Tab. this invariant ODE is entangled with different environmental factors, forming four environments, namely, damped, powered, spring, air.
2021b). The state X; [0., w/] E RR2 are the angle
{()). W {p). and Tc F. The underlying
dut
Table 4: ME-Pendulum ODEs with 0g ~ 2100, 4) and wp U(-1,0)
Environment
ODE for Og
ODE for We
Distribution of Parameters
Damped
d0, We
du dt
a 2 sin (0t) pur
Powered
Wt
dw a2 sin (0t) + P Wt
dt
dt dui
a2 sin (0t) por We
a U(100,2.0) 2.0)
Spring
d0:
= Wt
p ~~ ( 0.4)
dt
dt
Air
de = Wt
dwis dt
-a² sin (ft) plwy/lwr
dt
Invariant
d8, dt
= Wt
du' = - -2² sin (0z) dt
D.3. ME-Lotka-
Motivated by the Lotka- Volterra system (Ahmad 1993). the state X, = [(p, qd E R are the population of preys and predators at time i, where we have W = {ox 3, 7, 0) j. W
{0',,',,' 8'}. and Te = --- The underlying invariant ōpq 7g. As shown in Tab. these invariant ODEs are entangled in 4 environments, i.e.,
ODEs are dp fight resource, omnivore. The save environment ODEs simulate the decrease of food wastage along with the increase of predators. The fight environment ODEs simulate the decrease of hunting efficiency along with the increase of predators. The resource environment ODEs limit the increase rate of the prey population. In the omnivore environment ODEs, the predators are omnivores that can build the population without preys under certain resource limits.
dg di ap- 3pg, de
We plot the trajectories X in the training set, and the invariant trajectories X in the test set in Fig.
We plot the trajectories X in the training set, and the invariant trajectories X in the test set in Fig g
D.4. ME-SIREpidemic
In the SIREpidemic (Wang et al. 2021 system, the states X; = [St, 4,, Re] E R are the susceptible, infected, and recovered individuals at time t, respectively. In this adapted ME-SIREpidemic system, have W (B), W {y), and T. T 2
The underlying invariant ODEs are dr = - R di
S1
di
B S414R: di = O, where we only care about the S I
S1 dR
23

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Thiriing
Test
Figure 7: ME-Pendulum trajectories.
Table 5: ME--Lotka-Volterra ODEs with po ((0000, 2000) and go (4(00,200 20).
Environment
ODE for Pt
ODE for d:
Distribution of Parameters
Save
app
Bpq
B'py pg
10 10exp
dq
8pg
799
a, 2.4)
Fight
de op
3pg
dg dt
5pg
8' 8'pq
10 exp
--1)== YY
B, U(Ge 2, 1.2e
Resource
dp d
ap a'
2000
y²
Bpg
di 8pq
dt 20ry
7.7' ~ 4(0.48, 0.96)
Omnivore
dp
axp
3pq
(1
9.
74
6,6' u(4.8e 4, 9.66 4)
Invariant
de dt
ax 3pq
dg dt
8pg 74
Thaining Set. Weltttra-batterr
Inverient Test: ME Locks Wolss--oierrr
inerart
175
Figure 8: ME-Lotka-Voltee trajectories.
transformation relationship. As shown in Tab. 6 we introduce 4 environments, origin, enlarge, loop, negutive. These four environments describe four different models, where some of them are only for math modeling. The origin environment is the same as the original SIREpidemic model. The enlarge environment ODEs expand the epidemic range. The loop environment ODEs include deaths and second-fe time infections. The negative environment ODEs is a pure math model allowing negative numbers.
24

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Table 6: ME-SIREpidemic ODEs with So U9, 10), to ~ U(1,)) and Ro 0.
Environment
ODE for St
ODE for I
ODE for R₃
Distribution of Parameters
Origin
d8
B
dl B
71
dR 71
Enlarge
48
3 8i R +YI
81
B 81 71
dit I1
B ~ u(4,
Loop
d8
3+1+
7I YR
di
3
S+I+R 2
S1
dR II
YR
Y ~0
Negative
48
-5 5+1+R
df 0E
SI ylog!
dit
y log
B+I+R
Invariant
o2 d8 - 541+E
d di S41-8 R
dt 0
We plot the trajectories X in the training set, and the invariant trajectories X in the test set in Fig.
Thaining Set: demmi
Inarine1
klag
Laa
1.0
3.55
4.2
-200
Figure 9: ME-SIREpidemic trajectories.
E. Experimental Details
We conduct experiments on 800-sample training sets with training batch size of 32, which leads 25 iterations per epoch. For each run, optimize the neural network with 2,000 epochs, which is equivalent 50,000 iterations. Given fixed learning iterations, the learning rate is selected from ([e 4, le 3).
E.1. Baselines
In our experiments, we design four adapted baselines, since this task has never been explored before. The selection of baselines is based on the following two questions.
As invariant learning has been successfully applied in many representation learning tasks, can general invariant learning principle still work for invariant function learning?
Meta-learning techniques has been well designed solved problems in dynamical systems due to their quick adaptation
25

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
characteristics. However, there no evidence that the quick adaptable functions are the invariant function that shares across environments. Are they?
For the first question, since method is based on enforcing independence which shares similar philosophy domain adversarial neural network (Ganin et al. 2016), we consider other invariant methods going different ways. There are many methods well-known in the field of invariant learning, but considering our invariant function learning formulation, we found the discovery of invariant function requires independence that is different from the most typical "well performed across all environments" invariant learning requirements. Therefore, as validation of our guess, we adapt two widely used and known invariant learning methods, IRM (Arjovsky et al. 2019) and VREx (Krueger al. (021). These two techniques are directly applied our framework with the same architecture and their typical hyper-parameters searching spaces:
Xirm U(1e 2, le2)
Aprea U(1e le3)
As shown in our result 4 the results are not surprising, That means that the function can perform well across multiple environments is not the invariant function.
For the second question, our initial guess is that meta-learning is very sensitive to the distribution of the training set. certain pattern exists in multiple environments (not all), the meta-learning methods are prone to capture as part of the meta-function, which satisfies their quick adaptation goals. In our experiments, and CoDA (Kirchmeyer et al. 2022) as adapted baselines. CoDA is adapted since it uses hypernetwork as a full network adaptor which is similar test-time adaptation without trajectory encoder, leading to significant architecture differences. Therefore, we apply CoDA as meta-learning techniques focusing on its low-dimension (2-dimension) environment representation and regularization. MAML selected as the most typical meta-lcarning bascline, which does not require the use of hypernetwork, and only learns a meta function used to predict invariant trajectories. Their typical hyper-paramesn searching spaces are shown as follows.
choose MAML (Finn et al. 2017)
framework. However, CoDA is applied on coefficient-environmmen and requires
Aeoda U(1e 5, 1e 33
Amamt U(le 3, 1)
(Meta learning rate)
E. BASELINE DETAILS
For fair comparisons, tried our best make baselines as similar to architecture possible. So all the performance gains come from our IFL principle.
Architecture:
To better distinguish the baselines, let's denote our DIF framework without predictions (Figure and remove the Is MLP and branch) as DIF-Base.
MAML: Only Forecastor in the DIF-Base (with MAML optimization) since MAMI does meta-learning at the optimization level.
CoDA: DIF-Base and set the dimension of & = 2 (2 is the original paper setting).
IRM: DIF-Base. IRM regularization is at the loss level so no architecture change.
VREx: DIF-Base. VREx regularization is also at the loss level.
Inputs and Outputs:
MAMI. takes Xp and outputs the forecasting trajectory directly.
26

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
CoDA, IRM, VREx take Xp as input and output function representations & and مg. then finally output the forcasting trajectory. The same DIF.
Training:
MAML: MAML optimization on Forcaster.
CoDA, IRM, VREx the same as DIF.
Inference: (Decoding X for the baselines)
MAML: use the Forcaster with meta-parameters.
CoDA, IRM, VREx: the same as DIF: use the fe branch.
E.2. Disentanglement of Invariant Function Setup
E.2.1. ARCHITECTURE
Transformer. For the trajectory encoder in our hypernetwork, we apply a 6-layer 8-head 256-dimension FFN trans- former (Vaswani 2017) with frequency positional encoding (Gehring et al., 2017). We tried different architectures like GRUs (C14) but the transformer encoder can provide the best in-domain test performance easily. We also sweeped to the depth, width, and number of heads, and found that 6-layer 8-head 256-dimension FFN transformer is strong enough for ODE systems without making training difficult.
2014).
Function embedding. In the hidden function embedding space, we select the function embedding dimension to be 32 or 64, while these two selections perform quite similar. For the MLPs used to disentangle and decode hidden function embedding, we use 3-layer MLPs with ReLU as the activations. While for the decoder, the last layer projects the hidden function embedding to parameterized function space R" where m is the number of parameters in the derivative neural network.
Derivative function network. The derivative neural network is 4-layer or 5-layer MLP with width 16 32, which takes d.X. Rd This neural network is transformed be a functional in our implementation.
X, Rd input and output de
Discriminator. Our discriminator is 3 to 6 layer MLP with width 64 or 128. The size of discriminator can be easily chosen since the main goal of it is to discriminate the environment information in the hidden function space. Therefore, the simplest way to filter non-qualified discriminators is using the prediction entropy of Pa (e) f.), since f, should contain rich environment information, different to the prediction from f.. Our experiments also validate that the failure to distinguish f. will always cause the failure of invariant function discovery, which is natural, since the ad adversarial training is based on the optimal discriminators (Goodfellow et al. 2020).
Note that for all our MLPs, we apply one LayerNorm before each activation
E.2.2. TRAINING OBJECTIVES
We restate our three additional strategies here. Firstly, the adversarial training of f. will cause the loss of environment information in f., leading to the training difficulty of the discriminator, therefore, this discriminator is not only trained on f but also on f. with the same syper-parameter Aaie. i.e., ming.0 - _Ex,e~ log (e(faa)) Secondly, instead of using the
large fc and fe as the input of the discriminator, we input the the corresponding embeddings z. and ze. Thirdly, to avoid the use of numerical or neural integrator which causes long training time, we follow (Mouli et al. 2024) to fit derivatives only. That is, instead of using the inference forecaster p(X /he (Xp); Xo), we calculate the derivatives of X using f and fcr and replace the MSE over trajectory matrices with the MSE over derivative matrices. Note that this modification only eliminates the use of integrator for stability during training and thus does not affect our analysis and optimization goal.
We introduce our hyper-parameter searching space as follows.
*The code will released upon acceptance.
27

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
ME PeEdeedd
ME-Lotka Voleera
ME-SIREpiieemic
1.73
104
1.50
NRMSS
1.23
0.8
NRMSE
0.6
0.00
6.3
4
Orign
Origen
Origi
Figure 10: Ablation study on 3 DIF variants under 3 multi-environment ODE systems. For each pipeline, we provide model candidates with 50+ random hyper-parameter selections in their searching spaces, i.e., more than 450 models in the figure.
Xe -ulee 7, le - 4)
Asso u(le 1,
Nadu U(122, le6)
Aado de- Xado
The most critical hyper- syper-parameters are de and Ande which control the information overlap between f. and f. Conceptually, Ae controls the conditional mutual information (MI) maximization in our invariant function learning principle, while Aado enforces the independence constraint. The intensity of the independence enforcing Àadv is dependent on the intensity of MI maximization A thus, we set Åadv according to Ae, leading to Xado = A Xado
Adia is only discriminator training, which relatively trivial according to our discriminator descriptions in Appx. E.2.1
E.2.3. METRIC
Root mean square deviation (RMSE) is commonly used metric, but it suffers difficulties when comparing datasets with different value scales. Therefore, we normalize it using its standard deviation.
E X1I Std(X)
NRMSE
(26)
E.3. Software and Hardware
Our implementation is under the architecture of PyTorch (Paszke et al. 2019). The deployment environments are Ubuntu 20.04 with 48 Intel(R) Xeon(R) Silver, 4214R CPU @ 2.40GHz, 755GB RAM, and graphics cards NVIDIA RTX 2080Ti.
F. Supplementary Experiments
F.I. Ablation Study
As a complementary ablation study Sec. 5.5 the original model, namely, f pipeline and fe pipeline. The pipeline removes the discriminator and only output f to the forecasting, which neglects the disentanglement process. The fo pipeline prunes the ff output while maintaining the adversarial training process. As shown in Fig. [0] both f and fa pipelines fail to perform valid invariant function learning aligning with our theoretical results. Specifically, the í pipeline faces difficulties in extracting invariant functions without environment information. The unsatisfactory performance of the fe pipeline is attributed to the discriminator's training failure. This is because the training of the discriminator requires the capture environment information, but the elimination of the f part also removes the training of &e, the critical environment information captor. Therefore, the discriminator loses the most important environment information input, leading to training failure.
we train two models by eliminating two important components from
28

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
ME- PeE-Pend
MM-LotkaNVolterra
ME-SIREpidemicc
2.00
1.0
1.73
1.00
1.23
0.8
NRMSE
0.7
0.50
0.23
Figure 11: Trajectory input length study models trained with different training input length factors under 3 multi- environment ODE systems.
ME- ME-Pendulum
ME-Lotke-Vollerra
ME-SIREpidemicc
1.0
1.73
1.00
1.23
NRMSE
0.8
NAMSE
6.7
0.23
0.3
cons
cans
3 eats
cars
eavs
Figure 12: Environment analysis models trained with different numbers of training environments under 3 multi- environment ODE systems.
F22 Input Length and Environment Analysis
In order to ensure fairness, we fix the input length and the number of environments in experiments. However, it is also interesting to figure out the effects of the input trajectory length T. and the number of environments on the model performance. Fig. 11 shows the performance of DIF given different input length factor 4. where T, I. The results indicate the input length does not affect the performance significantly, where the only variances are attributed to the training difficulty of the transformer given different input lengths. Therefore, a shorter input length can perform slightly better given the same training steps.
f.
For the environment analysis, in addition to evaluations the full set of environments, benchmark model performance on datasets with three and two training environments. Specifically, select [Powered, Air, Spring] and [Powered, Air] for ME-Pendulum; [Save, Fight, Resource] and [Save, Fight] for ME-Lotka- Volter-o and [Negative, Origin, Enlarge] and INegative, Origin] for ME-SIREEpidemic. While not all possible environment combinations are evaluated, these selections provide intriguing insights. As illustrated in Fig. I2 changes in the set of environments weakly affect model performance on ME-Lotka- Volterra. For ME- Men-uend,,um, however, the inclusion of each additional environment consistently improves model performance. On ME-SIREpidemic, the performance boost observed with "3 envs" underscores the critical role of the environment Enlarge.
Two key observations regarding the ME-SIREEpieemic are worth noting. First, the average performance degradation on "4 envs" suggests a reduced focus on the important environment Enlarge due to the addition of the final environment Loop. Second, the improvement the best performance candidate demonstrates the additional benefits of the environment Loop. These findings illustrate that while adding environments can enhance the best possible discovery of invariant functions, also increases the average training complexity that may cause average performance degradations.
29

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
Table 7: Symbolic regression explanation comparisons.
ME-Pendulem
MB-Looka-Voltera
MI-SIEEpieeni
Method
NRMSE
SR Explanation
NRMSE
SR Explanation
NRMSE
SR Explanation
dS S,4, 1, cos (8)
d6
0.036 (8, +
de 9
P. + 0.63 (-0.022)
db 0.09
MAML
0,9704
de
0.6774
di
0.2673
d/ dt dR. sin (-00)))))
0.498, + (0.
duy (0.)
dg
0.48sin((4 s.68) (ax))
4-8
db -067
617
9r + 9/9
dt
d5.
S2 + sin 0.444
d9,
20p sin (0,- - wi++ 0.91)
+ + 0.074
[1.1 p.Pp) 0.54
d/.
3.58.
CoDA
0.9695
dui db
de
6, 0.20) 0.37)
0,7097
0.3184
S:
hor S
+.007
0.19
0.57
dt
(pt + Pv] (pt - go (-0.26) 0.66
dll.
0.029
di
4. + 4 16
d5.
1.7
d8g
de:
IRM
0,7042
dut
di
9,,0
0.6989
ol
0.9768
47,
sin
5. S S.
-0.1
di
dR.
0.30
0.0218
dt
d8c
0.103)
d9,
dm 0.032
VREX
0.7274
de
wky 0..92
0.6877
dt
0.4652
d/c
5,8
0.078
deh
8 4
8,
a (-1.1)
dt
de
dR. 0.070
sin (3)
d
d5 -1.5,
d9:
Ours
0,3561
dt
0.6194
dt
1.25-4p
0. 0.388
0.0652
d/. 0.428,1
d2
-0.90 0.977 sin (9))
dt 4.1
0.300
dR. 0.0088
dS.
8,4,
d8:
dy
St
Jt Re
GT
NAN
de
NAN
oph
Apeak
NAN
5,1.
dur
o7
2 sln (0.)
dg. de 6pen
79
dRt
1,
F33 Symbolic Regression Explanation Comparisons
To further evaluate the performance differences between the proposed method and the baselines, we apply PySR to the four baseline methods and obtain analytical explanations, as summarized in Tab. 7 The symbolic regression results provide an intuitive understanding of performance relation to different NRMSE values. Specifically, when the NRMSE approaches the resulting explanations are largely meaningless. As the NRMSE decreases to around 0.7, the explanations become more interpretable but may sometimes converge to oversimplified expressions, such as the IRM and VREx results on Le-Lotka-Volteree When the NRMSE approaches zero, the expressions become more reasonable but are not always ideal due to the inherent limitations of PySR. This suggests that strong model performance does not necessarily guarantee high-quality explanations, highlighting the performance constraints of the explainer (PySR).
F4. Extra Visualizations
In this section, we present visualization comparisons for ME-Lotka-Volten (Fig. 13 and MI-SIIEpidemic (Fig. 14p. The results for ME-SSRREpidemic closely align with its quantitative findings. For the more challenging task of ME-LLtta- Volteraa, our method's predicted trajectories remain closer to the ground truth. In the X predictions, where most methods fail, our predicted trajectory has turning points closest to the ground truth terms of timing, although there are deviations in magnitude (Fig. [3b]. The complexity of the ME-Lotka- Volterr task arises from several factors, including the introduction of exponential functions within environments, the distribution of environments, and the limited number of samples in each environment. Addressing these challenges requires carefully designed benchmarks by domain experts, which we will discuss further in the limitations section.
G. Efficient Hypernetwork Implementation
30

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
0.3
35
4.00
VITES
0.6
IBM
DAPS
2.3
IRM VRTY
03
352
VEEX
3.23
MANE
0 3.0
0.5
0"
3.00
Craaad TrNEA
1.4
Uroaed X
1234
2.75
0.3
OLomrd 7osh
(a) predictions using f
(b) X predictions using fa
Figure 13: Visualization of trajectory predictions on VottertaaVotttrraa
Dre
Mide:
(a) X predictions using f
(b) X" predictions using fc
Figure 14: Visualization of trajectory predictions on ME-SIREpidemie
One of the major challenges that limits the usage of hypernetworks is the implementation complexity. Most current implementations requires either re-implementing basic neural networks (von Oswald et al. 2020) or assigning predicted weights to the main function (forecaster) one by one for each forward pass (Ortiz et al. 2023 Sudhakaran [2022 Kirchmeyer et al. 2022). To overcome these issues, we propose Reference- -baee hypernetwork implementation technique that uses pure Py Torch without introducing any new modules or CUDA kernels. Our proposed technique does not require reassigning weights for each sample in one forward pass, i.e., for any continuous N training iterations with batch size of B and a forecaster with M parameter variables, our computation complexity is O(N + BM), instead of ((BMN) as previous implementations.
Specifically, for every forward pass, we create function parameter vector buffer E RT with fixed storage space, instead of reshaping and assigning the predicted function parameters with complexity O(BM). As shown in Fig. 15 we consider the derivative neural network parameter variables as storage space pointers, i.e., the network stores references instead of matrices. The fractions of function parameter vector buffer are pointed by these pointers; thus, once the buffer's values change by the predicted function parameters, egg., f. the derivative network's parameters will changed automatically without any assignment operators. To maintain the buffer's fixed storage space, several in-place operations are applied to maintains computational graphs and gradients.
G.1. Efficiency Comparisons
To evaluate the efficiency of our hypernetwork implementation, we compare against several common implementation approaches. Specifically, measure the forward pass time of our model over 200 continuous iterations in training
Table 8: Hypernetwork implementation efficiency comparisons
Implementation
Vectorization
Copy
Reference
First Step Time (s)
Avg Time + Std (s)
Speedup
Non-vectorized
0.2466
0.1818 + 0.0601
Ix
Module-based
/
x
0.1768
0.1513 + 0.0737
1.2x
Functional-based
X
x
0,2013
0.0198 + 0.0007
9.2x
Ours
X
/
0.1805
0.0108 + 0.0006
16.8x
31

---

Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
mode, recording both the time for the first iteration and the average time for the subsequent iterations. While the first iteration typically takes similar amount of time across all implementations, their performance diverges significantly in the subsequent iterations. As shown in Tab. 8 the Non-vectorized implementation represents methods that do not vectorize the derivative function and therefore must run different derivative functions sequentially. Approaches like CoDA (Kirchmeyer et al. 2022) attempt to vectorize the model by employing group-hased convolution networks. However, these module-based implementations rely on stateful PyTorch modules, requiring the derivative function module to be replicated during each forward pass, which slows down the process. While these Module- implementations offer slight improvement over non-vectorized methods due to the vectorization benefits, the performance gain is limited.
In contrast, vectorized Functional-basee implementation leverages PyTorch's s functional methods, achieving 9.2x speedup by avoiding the overhead associated with stateful modules. Note that vectorizing hypernetworks using libraries such as hypnettorch (von Oswald et al. 2020) can deliver similar speedups. Finally, our Reference- based implementation, which eliminates parameter assignment after the first iteration, nearly doubles the forward pass speed (16.8x) compared to implementations that require such assignments. Notably, this optimization remains applicable for potential future CUDA-bass parallel hypernetwork implementa- tions.
Parameter buffer
R"
X
Reference pointar
Parameter pointers
copy
dX di
Figure 15: Reference- hyper-
network implementation.
32


Reviews:
[{"review_id": "Ys84C6AZOU", "reviewer": "Reviewer_pHQv", "summary": "The authors proposed the task of learning invariant functions in dynamical systems: considering the ODE equation $\\frac{\\mathrm{d}\\mathbf{x}}{\\mathrm{d}t} = f(\\mathbf{x})$ of the system evolution as a combination of the invariant function $f_c$ corresponding to the inherent properties of the system itself and the function $f_e$ caused by the environment, and learning the common part $f_c$ from the trajectory data collected in different environments. To solve this problem, the authors proposed the DIF method, which uses a meta neural network to generate parameters of two neural networks $\\hat{f}$ and $\\hat{f}_c$ based on a single trajectory $X$ collected in a specific environment to fit the governing ODE equation $f$ and the invariant function $f_c$. DIF minimizes the MSE between the system's evolution trajectory governing by $\\hat{f}_c$ and the actual evolution trajectory $X$ (equivalent to maximizing the mutual information of $\\hat{f}_c$ and $f$), and uses adversarial training to make a discriminator unable to infer information about the environment from $\\hat{f}_c$, so that the learned $\\hat{f}_c$ becomes an invariant function that describes the evolutionary properties of the system itself and is invariant to the environment. The authors conducted experiments on three systems: Penduium, Lotka-Volterra, and SIR. They considered four variants of each system under different environments and generated 200 trajectories for each variant using different parameters and initial values. DIF learns the system's invariant function $\\hat{f}_c$ from these trajectories. The evolution trajectory governing by $\\hat{f}_c$ is consistent with the evolution trajectory governing by the ground truth $f_c$ (NRMSE < 1.0, suggesting positive correlation). The authors further extract the symbolic expression of $\\hat{f}_c$ through symbolic regression, finding that it is similar to the ground truth invariant function, proving the ability of the DIF method to learn invariant functions in dynamic systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad you recognized our theoretical framework and experimental validation. We address your concerns point-by-point as follows.\n\n## Practicability in real-world scenarios\n\nThank you for this question.  While we focus on establishing theoretical foundations in this work, we believe our method has significant real-world potential. In principle, the methodology, particularly the disentanglement via adversarial training and mutual information maximization, is applicable to real-world scenarios where environmental variations are present. Invariant function learning can extract fundamental dynamical laws from noisy, environment-influenced data, a critical challenge in scientific discovery. Application areas include:\n\n1. Extracting physical laws from video data where environment factors (e.g., lighting, camera position) complicate analysis\n2. Learning generalizable physics for simulation and control across varying conditions\n3. Contributing to foundation models in physics by identifying invariant mechanisms\n\nNote that some of these applications might not belong to invariant function learning but fall back to general invariant learning, which is not our target. PDE scenarios that need invariant function learning, are in our future work since much work needs to be done before being able to transfer to PDEs described in Appendix B Q1.\n\n## Theoretical claims and noise assumptions\n\nWe acknowledge that Appendix C.3 assumes Gaussian white noise. While this is a common starting point in theoretical analysis, we agree this is a limitation. Future work will extend our framework to other noise scenarios or relax this assumption.\n\n## High-dimensional systems with graph structure\n\nThank you for the suggestions and discussions on graph-structured physical mechanisms as in Cranmer et al. (2020) and Shi et al. (2022). We agree these are relevant to extending our work. Our current focus was establishing the basic framework for invariant function learning in ODE systems, but we acknowledge that discussing the approach to high-dimensional or graph-structured settings is worthwhile. In the revised paper, we will discuss these limitations and outline potential directions for adapting our method to such complex scenarios.\n\n## Utility Compared to Alternative Approaches\n\nThank you for your comparison with direct neural network fitting and symbolic regression approaches. Our method targets extracting invariant dynamics. Regarding alternative strategies like finding symbolic expressions in different environments and then obtaining the same part $f_c$ through *observing*, the most difficult part is *observing*. The reason is direct: the fitting of f tends to capture both invariant and environment-specific aspects, which leads to spurious correlations so that the final equation forms from different environments **seem significantly different**. That is to say, after extracting equations from different environments, **it is challenging to find the common part directly** since the common parts are blended with environment parts and **look like not disentangable**.\n\nOur DIF method explicitly enforces the separation of invariant dynamics from environment-specific factors, providing a more reliable basis for scientific discovery. This advantage becomes especially important in scenarios with complex environmental effects or when specific invariant mechanisms need to be identified. We will expand on this distinction in the revised manuscript.\n\n## Real-world datasets\n\nWe agree that real-world validation would strengthen our paper. Our focus in this initial work was establishing theoretical foundations and validating on controlled systems. Testing on real-world data is the next step we discussed in the Appendix and we're actively pursuing this direction.\n\n## Question on X_p vs X\n\nRegarding the specific question (Q1): Your understanding is mostly correct. We distinguish between $X_p$ (past observations before time $T_c$) and $X$ (full trajectory) because the forecasting task takes $X_p$ as input $X$ as output. Similar to $X\\rightarrow Y$ in general prediction tasks, the forecasting task does $X_p \\rightarrow X$. In training, $X$ works as \"label $Y$\" for loss, while in inference time, $X$ after $T_c$ is not available. Using the whole trajectory $X$ as $X_p$ would blur the line between training and testing data. We will clarify this point further in the revised manuscript.\n\nWe hope these responses address your concerns, and we thank you for your constructive feedback.", "claims_and_evidence": "Mostly, the structural causal model drawn by the authors (Figure2) illustrates the concept of invariant functions very well, and a complete mathematical proof for the training scheme used is given in the appendix. The effectiveness of the proposed DIF method is also demonstrated experimentally on three systems.\n\nHowever, I have doubts regarding the practicability of the proposed method in real-world scenarios.", "methods_and_evaluation_criteria": "Yes, although the authors focus on a new problem and therefore have no existing dataset, they construct three datasets based on the Penduium, Lotka-Volterra, and SIR systems that are consistent with the invariant function learning task they describe.", "theoretical_claims": "I read the theoretical proof in Appendix C, and there is no obvious problem with the derivation process. However, the proof in Appendix C.3 relies on the assumption of Gaussian white noise, that is, the evolution trajectory is the result of adding Gaussian white noise to the noise-free evolution trajectory. This may render the derivation untenable when evolutionary trajectories contain colored noise that is more common in the real world. This is particularly concerning given that the authors did not present experiments on real-world datasets.", "experimental_designs_or_analyses": "Yes, the authors' experimental design effectively illustrates the ability of DIF to learn invariant functions of the system itself from evolutionary trajectories collected in different environments.", "relation_to_broader_scientific_literature": "The author proposed for the first time the task of invariant function learning in dynamic systems. This task is related to Meta-Learning and invariant function learning. However, existing invariant function learning methods are not applicable to dynamical systems that evolve over time; and it is questionable whether the functions learned by Meta-Learning that can quickly fit different environments are invariant functions.", "essential_references_not_discussed": "I think authors should discuss possible limitations of the proposed method in high dimensional systems with graph structure, as in following papers.\n\n1. Cranmer, Miles, et al. \"Discovering symbolic models from deep learning with inductive biases.\" Advances in neural information processing systems 33 (2020): 17429-17442.\n\n2. Shi, Hongzhi, et al. \"Learning symbolic models for graph-structured physical mechanism.\" The Eleventh International Conference on Learning Representations. 2022.", "other_strengths_and_weaknesses": "**Strengths**: The authors proposed the task of learning invariant functions in dynamical systems and constructed a solid theoretical framework and a novel DIF method for this task. Although this task seems rather challenging, the learned invariant functions are surprisingly consistent with the real results in terms of both evolving behavior and symbolic nature.\n\n**Weakness**: Although the authors provide a discussion of possible applications of the method in Appendix B (Q5), it is difficult to imagine any specific scenarios for this task. If we want to get a predictive model that predicts system behavior, directly using a neural network to fit $f$ (rather than $f_c$) seems to be a better choice; if we want to get a symbolic model that explains the system behavior, directly using symbolic regression methods to find $f$'s symbolic expression in different environments and then obtaining the same part $f_c$ through observing seems to be a better choice.", "other_comments_or_suggestions": "Considering that the authors focus on a new task that learns invariant functions of dynamical systems, it would be better to provide experiments based on real-world datasets, which can help readers establish an understanding of the application scenarios and importance of this task, and, on the other hand, better verify the capabilities of the proposed DIF method.\n\nAlso, it is better to discuss its applicability in much complex scenarios like networked complex systems (as in my provided references).", "questions_for_authors": "**Q1**. If I understand correctly, $X$ is the observed evolution trajectory, and $X_p$ is the part of $X$ before $T_c$. In training, $X_p$ is used as the input of the model, and $X$ is used to calculate the loss function. Why do we need to distinguish between $X_p$ and $X$? Is it feasible to not distinguish between them and take the whole $X$ as $X_p$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9qS2ogcVs", "reviewer": "Reviewer_cD7J", "summary": "This paper introduces an approach to learning invariant functions in dynamical systems (governed by ODEs) in different environments. The authors propose a method called Disentanglement of Invariant Functions (DIF), which aims to discover intrinsic dynamics across multiple environments while avoiding environment-specific mechanisms. The approach is an encoder-decoder hypernetwork to disentangle invariant functions from environment-specific dynamics. The authors have shown that the probabilistic information-maximization objective can be implemented using a simple easy-to-implement MSE loss function. The method is evaluated against meta-learning and invariant learning baselines, demonstrating its effectiveness.\n\n## update after rebuttal\nThank you for the detailed response and for providing additional context regarding your design choices and theoretical connections. While I appreciate the clarifications, my initial concerns regarding the generalization and broader applicability of the work remain. As these issues are still relevant to the overall impact, I have decided to keep my original evaluation unchanged at a 3 (Weak Accept). I recognize the potential of the work, but feel that these concerns need further attention.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and the acknowledgment of the novelty and effectiveness. Below, we address your concerns point by point.\n\n## Limited experimental setup\n\nThank you for raising this concern. Our work represents the first step in invariant function learning, introducing a new paradigm for scientific discovery. The focus on low-dimensional systems was intentional for three reasons:\n\n1. **Conceptual clarity:** Low-dimensional systems allow us to clearly demonstrate the core principles of invariant function learning without obscuring them with implementation details for high-dimensional data.\n2. **Methodological validation:** As the first work on invariant function learning, we needed to establish baseline performance on well-understood systems where ground truth is available.\n3. **Benchmark contribution:** We've constructed three multi-environment datasets (ME-Pendulum, ME-Lotka-Volterra, and ME-SIREpidemic) that will serve as important benchmarks for future research.\n\nWe fully agree that extending to high-dimensional real-world examples is an important direction, as we acknowledge in our limitations section (Section 6). The current work lays the theoretical and methodological foundation for these future extensions.\n\n## Scope limitation to ODE systems\n\nThank you for this important question. As we discussed in **Appendix B**, we acknowledge that our method is currently designed for ODE systems. This choice was driven by three key challenges:\n\n1. **Lack of multi-environment PDE datasets:** Unlike domain adaptation tasks, multi-environment datasets for PDEs aren't readily available and require domain-specific expertise to construct. \n2. **PDE-specific technical challenges:** PDEs introduce multi-scale and multi-resolution problems requiring specialized techniques for stability and computational efficiency.\n3. **Interpretation challenges:** Our current approach uses symbolic regression for interpretability, which doesn't extend naturally to PDE systems.\n\nOur future work will tackle these challenges by: (1) collaborating with domain experts to develop multi-environment PDE benchmarks, (2) incorporating specialized PDE-handling techniques from neural operators literature, and (3) developing new interpretability methods for invariant function learning in PDEs.\n\n## Connection to causal mechanisms\n\nWe agree that there is indeed a strong connection to the independent causal mechanism (ICM) assumption/principle. Our work can be viewed as discovering ICM specifically to function spaces in dynamical systems.\n\nNote that \"SCMs are underpinned by the assumption of independent causal mechanisms, a fundamental premise in causality. Intuitively, this supposition holds that causal correlations in SCMs are stable, independent mechanisms akin to unchanging physical laws, rendering these causal mechanisms generalizable.\"\n\n**ICM is the basis of Jonas's invariant causal predictor and IRM**. When we use SCM in Section 2.2.1, we already adopt the ICM assumption, as in the SCM, we're assuming that invariant mechanisms (fc) and environment-specific mechanisms (fe) operate independently and can be disentangled. From the ICM perspective, our optimization goal is to identify the ICM showing in our SCM.\n\nWe acknowledge that we could have made this connection more explicit in our related work, and we appreciate the opportunity to clarify it here. In our revised version, we would enhance our discussion of this connection in the related work section.\n\n## Novelty limitations and related work\n\n> Although the experiments show the effectiveness of the proposed method, the use of information maximization to disentangle causal mechanisms has been used in various contexts before.\n\nWe acknowledge that information maximization for disentangling causal mechanisms has precedents in the literature. Our contribution lies in:\n\n1. **Novel task formulation:** We introduce invariant function learning as a new task for scientific discovery (Section 2.2).\n2. **Function space application:** While previous work has applied similar principles to fixed representations, we extend them to function spaces where invariance must be maintained across all possible states.\n3. **Theoretical guarantees:** Our work provides theoretical guarantees for invariant function discovery (Theorem 3.1) specifically adapted to the dynamical systems context.\n4. **Empirical validation:** Our experimental results (Section 5.4) demonstrate that general invariant learning principles like IRM and VREx do not perform well on our task, highlighting the need for our specialized approach.\n\nThese points, we believe, substantiate the novelty and significance of our approach.\n\n## Regarding essential references\n\nThank you for this valuable suggestion. We agree that enhancing our discussion of connections to ICM and ICA literature would strengthen the paper. In the revised versions, we would expand Section 4 (Related Work) to include them.", "claims_and_evidence": "The effectiveness of the method could be limited given the experimental setup.", "methods_and_evaluation_criteria": "Yes, but mostly limited to low-dimensional simulated data.", "theoretical_claims": "The connection between information-maximization and MSE loss is a known theoretical result and makes intuitive sense. I did not check the details of the proof in the appendix though.", "experimental_designs_or_analyses": "The design of the experiments sounds logical but lacks high-dimensional real-world experiments to support the effectiveness of the method in more challenging domains.", "relation_to_broader_scientific_literature": "Although the design of the hyper-network may be novel in discovering dynamical systems, it is not so novel in ML literature as there have been methods to use prediction loss as a proxy for information-maximization and also representing functions as weights of a NN.", "essential_references_not_discussed": "Connection to Independent Causal Mechanism and Independent Component Analysis.\nThere is a vast literature on both topics. I think this work should also be discussed in relation to these concepts.", "other_strengths_and_weaknesses": "## Strengths\n- Novelty: The paper presents a new perspective on invariant function learning, focusing on disentangling invariant dynamics from environment-specific factors.\n\n- Methodology: The proposed hypernetwork which is based on information-maximization objectives derived from the causal-graph is relatively innovative in this context and in line with gradient-based learning which suggests better scalability compared with Bayesian inference methods.\n\n- Effectiveness: Quantitative comparisons show that the proposed method outperforms existing meta-learning and invariant learning techniques on the simulated data benchmarks.\n\n## Weaknesses\n\n- Scope Limitation: The method is currently limited to ODE systems and seems not directly applicable to PDE systems which govern a large class of dynamical systems.\n\n- Generalization: The experiments are all low-dimensional and with relatively simple dynamics and simulated data. High-dimensional real-world examples (e.g. from robotics, cell biology) could better illustrate the effectiveness of this method.\n\n- Novelty limitation: Although the experiments show the effectiveness of the proposed method, the use of information maximization to disentangle causal mechanisms has been used in various contexts before.", "other_comments_or_suggestions": "-", "questions_for_authors": "- Could you elaborate on the limitations of extending your method to PDE systems and how you plan to address them in future work?\n\n- Based on the definition of invariant and environment-specific dynamics used in the paper, it seems there is a strong connection to the independent causal mechanism principle. Can you elaborate if you can recognize such connection?\n\n- It seems all evaluations are done with simulated data. The assessment of the potential of the work would be easier if the authors could evaluate with real-world high-dimensional examples both in terms of scalability (higher dimensions) and also in terms of dealing with more complex dynamics.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0hbUcZ7eG", "reviewer": "Reviewer_k51o", "summary": "This paper proposes a method for discovering invariant functions underlying dynamical systems governed by Ordinary Differential Equations (ODEs). The key claim is that different environments modify the observed system, but an invariant function can be disentangled and recovered to represent the core governing dynamics. The authors introduce a framework called Disentanglement of Invariant Functions (DIF), which uses an encoder-decoder hypernetwork to separate invariant functions from environment-specific dynamics. The paper also presents a causal graph-based formulation to formalize the problem and proposes an information-theoretic principle to enforce invariance.\n\nTo validate their approach, the authors conduct experiments on three multi-environment ODE datasets, comparing DIF against baseline methods such as meta-learning approaches and traditional invariant learning techniques. The results reportedly demonstrate that DIF is more effective in recovering the underlying physical laws across environments and performs well in symbolic regression tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your feedback and would like to address your concerns point by point.\n\n## Definition and identifiability of invariant function\n\n> The definition of an invariant function is vague...\n\nWe thank the reviewer for raising this point.\n\n- **Rigorous definition**: Our invariant function is not defined arbitrarily; it is rigorously grounded in causality and formally specified within our Structural Causal Model (SCM) (see **Section 2.2.1, Figure 2, and Appx. C.1**). Specifically, the invariant function fc is the function generated by the structural equation and exogenous variable c, which represents the core mechanism shared across all environments. This causal definition distinguishes our approach from ad-hoc solutions. \n\n- **Example pitfall**: We respectfully argue that the reviewer’s example oversimplifies the problem by focusing on a single realization, however, our treatment considers the invariant function as a random variable, a distribution that must be learned, not a specific instance.\n\n- **A more proper example**: If we have $\\mathbf{f}_1=\\bm{\\alpha} x + \\bm{\\beta}$ and $\\mathbf{f}_2=\\bm{\\alpha} (x + 1) + \\bm{\\rho}=\\bm{\\alpha} x + (\\bm{\\alpha} + \\bm{\\rho})$, then since $f_c \\perp e$, we find the decision boundary for the $e$ prediction, which determines by the distribution differences between $\\bm{\\beta}$ and $\\bm{\\alpha} + \\bm{\\rho}$, not by the distribution of $\\bm{\\alpha} x$ (invariant). Note that the three functions the reviewer provide can be sampled from both $\\mathbf{f}_1$ and $\\mathbf{f}_2$, **illustrating the pitfall of analysing in instance level**. This example is not perfect. For rigorous understanding, please resort to our SCM and information theory based proofs. Furthermore, we invite the reviewer to refer to **Appendix B (Q2)** and related works in invariant learning for additional insights.\n\n- **Community acknowledgment**: The invariant function is defined rigorously in a statistically significant way using SCM. We like to share that using SCM to define invariant representation/mechanism has been a mature knowledge in invariant representation learning community. \n\n> There is no strong theoretical justification for why the invariant function is identifiable in all cases.\n\n- **Identifiability: Sufficiency & necessity**: In Section C.2 of the appendix, we provide a complete proof of the **existence** and **uniqueness** of the solution to our optimization problem. The proof demonstrates that our invariant function learning principle guarantees identifiability under the causal assumptions.\n\n> I do not fully understand some theoretical claims in this paper, Theorem 3.1 (and its proof) in particular.\n\nTheorem 3.1 provides the optimization principle to sufficiently and necessarily discover the function random variable defined in our SCM.\n\nNote that our definition and theoretical proof have been acknowledged by other reviewers:\n\n**Reviewer pHQv** stated: \n\n- \"structural causal model drawn by the authors (Figure2) **illustrates the concept of invariant functions very well**\" \n- \"I **read the theoretical proof** in Appendix C, and there is no obvious problem with the derivation process.\"\n\n**Reviewer cD7J**: remarked on the solid theoretical framework and highlighted the connection to Independent Causal Mechanism, underscoring that invariant learning is a **well-established area**.\n\n**Reviewer T91b** confirmed \"I have **checked the main theoretical claims** (3.1, 3.2 and 3.3) and I am happy with their **soundness**.\"\n\nDue to space limitations, for further context, we recommend reviewing the fundamentals of invariant learning and the role of SCM in invariant learning listed in our essential related works.\n\n## Experimental Design and Generalization\n\n> One key concern is whether the method generalizes well to settings where environmental changes are more drastic. ... The \"powered\" environment would possibly break the typical behavior of the pendulum and fling the mass over the top.\n\n**Theoretically**: The alterations in parameters (α and ρ) do not undermine the structure of our SCM; hence, our method remains generalizable. \n\n**Empirically**: Our experiments in the \"powered\" environment **do fling the mass over the top**, and the results shown are still robust, suggesting it **can handle** increasingly drastic changes.\n\n## Writing and Organization\n\nWe appreciate this feedback and will improve the writing in our revision. We will:\n\n1. Address redundancy and improve grammar\n2. Enhance coherence, particularly in Section 2.2\n3. Fix notation issues.\n4. Clearly define all variables before use\n5. Reorganize the explanation of challenges and solutions for improved flow\n\n## Related Works\n\nThank you for highlighting these important references. While we did mention PySR and symbolic regression in the Appendix, we agree that SINDy and genetic programming-based methods deserve more attention in the main text. We will expand Section 4 (Related Work) to include them.", "claims_and_evidence": "I am doubtful towards the main claims of this paper.\n* **Claim: An invariant function exists that captures the common dynamics across environments.**\n    * The definition of an invariant function is vague. At first thought, it represents the \"common parts\" of different governing equations, it does not specify what constitutes \"common.\" As an analogy, what is considered the invariant function for $f_1(x) = x,\\ f_2(x) = x+1,\\ f_3(x)=x+2$? Should it be $x$, due to its simplicity, or $x+1$ because it is the average of all functions? Anyways, how does one rigorously define this commonality beyond intuition? The provided examples, such as variations of a pendulum's equation, seem to assume that a clear invariant function exists, but this assumption is not justified theoretically. If environments are too diverse, there may be no meaningful invariant function.\n* **Claim: The method can successfully identify the invariant function across environments.**\n    * There is no strong theoretical justification for why the invariant function is identifiable in all cases. If the environments are drastically different, could it be possible that the method learns an arbitrary function rather than the true underlying invariant one? Moreover, since there seems to be no clear definition of the \"true\" invariant function, as pointed out earlier, how do we know the learned invariant function is correct?", "methods_and_evaluation_criteria": "* The paper constructs multi-environment ODE datasets to evaluate the method. The datasets are reasonable choices for testing the framework since they include variations of well-known physical systems.\n* The performance is measured in terms of error in predicting future states for both the observed trajectories and the invariant trajectories. These metrics align with the goal of recovering governing equations, but in terms of verifying the \"correctness\" of the invariant functions, the authors assume the knowledge of a \"ground truth\" invariant function. However, as discussed earlier, it is unclear to me why these specific equations are preferred over others as the invariant function.", "theoretical_claims": "I do not fully understand some theoretical claims in this paper, Theorem 3.1 (and its proof) in particular. What defines the \"true invariant function\"? From the statement I understand that we have the solution to an optimization problem, which can be mapped to a function $f_c$. Then what are we trying to prove? Is it some property of this function? Or is it the existence and uniqueness of this solution, as seemingly suggested by the proof? If it is the latter, I don't understand what this theorem is trying to establish.", "experimental_designs_or_analyses": "* The experimental results indicate that DIF generally achieves lower error and better symbolic regression outcomes than baselines.\n* One key concern is whether the method generalizes well to settings where environmental changes are more drastic. For example, what would happen if you make $\\alpha$ smaller and $\\rho$ larger in the pendulum experiment? The \"powered\" environment would possibly break the typical behavior of the pendulum and fling the mass over the top.", "relation_to_broader_scientific_literature": "* The work is related to previous studies on invariant learning (e.g., Arjovsky et al., 2019; Rosenfeld et al., 2020), but many prior work focuses on categorical settings rather than function spaces. The authors attempt to extend these ideas to dynamical systems, which is a reasonable direction.\n* There are abundant works on governing equation discovery for ODE systems, deriving from SINDy (Brunton et al., 2016), which the authors should have discussed.\n* Apart from SINDy sparse regression, genetic programming-based methods are also widely applied to this task. The authors did mention related works such as PySR (Cranmer, 2023) in the Appendix, but I feel it should also be included in the related works section in the main text.\n\n\n### References\n* Brunton, Steven L., Joshua L. Proctor, and J. Nathan Kutz. \"Discovering governing equations from data by sparse identification of nonlinear dynamical systems.\" Proceedings of the national academy of sciences 113.15 (2016): 3932-3937.\n* Cranmer, Miles. \"Interpretable machine learning for science with PySR and SymbolicRegression. jl.\" arXiv preprint arXiv:2305.01582 (2023).", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "The writing of this paper can be significantly improved. First of all, it is too verbose, and I'd suggest the authors run a grammar check using Grammarly to reduce redundancy. The paper also lacks coherence at some point. For example, in Section 2.2, the authors first identify two challenges in L147-161, but I don't understand how the paragraphs describing solutions in Section 2.2.1 correspond to these two challenges.\n\nThen, in terms of math writing, there are many instances of missing definitions, incorrect notations, and confusing statements. Besides the lack of clear definition of an invariant function already mentioned above, other examples include:\n* $\\mathbf X_0$ is referred to before definition.\n* When expressing a function mapping, the common practice is to write $f: X \\to Y, x \\mapsto f(x)$ instead of $f: X \\mapsto Y$.\n* In Thm 3.1, why \"given the predicted function random variable $\\hat f_c$? It is not used anywhere in the theorem or the proof.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See previous sections.\n\n## Post-rebuttal\n\nOne of my main concerns was the validity of the theoretical claims (Theorem 3.1, in particular) made in the paper. After the discussion with the authors, I believe it is mostly a clarity issue and the claims are mostly valid. Concretely:\n\n* Thm 3.1 is essentially saying that $f_c$ is the unique solution to the optimization problem $\\arg\\max_{f'} I(f';f | X_0)$ s.t. $f' \\perp e$. It makes sense when all the $h$ and $\\theta_c$ are removed from the statement and its proof.\n* The original statement in Thm 3.1 is misleading and also (slightly) incorrect. While a unique solution $f_c$ always exists, there is no guarantee that the function class of all possible $h$, which depends on the implementation of the hypernetwork, can approximate the true $f_c$. That being said, this is fine in practice. The authors just need to rewrite the theorem and make a comment about this gap between theory and realization.\n\nOverall, I think this paper has some valid contributions, though marred by the lackluster presentation. It could be accepted should the authors improve the writing and fix the clarity issues. I have updated my score to reflect this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "arOCr2T1kV", "reviewer": "Reviewer_T91b", "summary": "This paper propose a method to learn ODE based dynamical systems from observed sequences. The main feature of the proposed method which sets it apart from prior work is that it can learn invariant functions that can be reused across different environments, effectively disentangling general and reusable functions from environment-specific functions. This is motivated by the general context of automating scientific discoveries from observations. The authors propose to achieve this disentanglement by learning a function embedding which is maximally informative about the observed sequence while remaining independent on which specific environment the sequence is from. The paper offers a theoretical account of why this approach works and empirically verify their claims with three common dynamical systems that are modified to the multi-environment setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. We will address them point by point below.\n\n## Drastic and non-linear changes across function environments\n\n> What would happen if changes across function environments were more drastic and non-linear? For example, changing f(theta) = sin(theta) to f(theta) = sin(theta)^2.\n\nThank you for this insightful question. In short, drastic changes are actually good for invariance discovery, but non-linear changes are much more sophisticated to analyze.\n\n**Drastic changes**: According to our initial experiments, larger environmental differences will lead to better results. The experimental results shown in the paper indeed come with decent environment differences. For example, in the \"powered\" pendulum environment, the mass is flung over the top in many trajectories (Figure 7: upper left one).\n\n**Non-linear changes**: In most optimization problems, expanding from linear to non-linear is challenging. For example, the invariant learning pioneer work IRM can only prove that the invariant principle works under the linear scenario and one SCM. Therefore, we don't expect our method to work *directly* for all non-linear scenarios. We provide analyses and open discussions around it and hope you find it interesting and useful.\n\n- First, we consider the theory basis, the SCM in Appendix C.1. In the SCM, $f:=g\\_{comp}(f\\_c, f\\_e)$, while $f\\_e$ can be different across $e$, $g\\_{comp}$ is shared. That means, if we consider $f(theta) = sin(theta)^2=sin(theta) * sin(theta)$ as $f:=f\\_c * f\\_e$, then the SCM remains valid if all the environments obey $f:=f\\_c*f\\_e$. If not all environments share the same function composition process, extended SCMs or extended $g\\_{comp}$ will be required to be designed.\n- Second, if SCM remains valid, the next challenge is the reverse decomposition (red dash arrows in Figure 6). In the current hyper-network implementation, given z_c denotes sin(theta), z_e denotes sin(theta), letting z_c + z_e denote sin(theta)^2 in the function representation space is a learning challenge, which may require much deeper networks or extra techniques. For example, we may choose to learn in the space of the derivative of sin(theta)^2 to alleviate the learning challenge.\n- Finally, in the real world, many effects are composed additively or can be transformed to be composed additively like forces, so we believe our method is applicable in many cases.\n\n## Baseline implementation details\n\n> Please clarify the baselines. It is unclear how the baselines are implemented, what are the architectures, what are the input and outputs, and do they also have separate embeddings z_c and z_e?\"\n\nWe appreciate this important point about baseline clarification. In the revised paper, we will provide a detailed explanation of the baseline implementations:\n\nBasically, for fair comparisons, we tried our best to make baselines as similar to our architecture as possible. So all the performance gains come from our IFL principle.\n\n**Architecture**\n\nTo better distinguish the baselines, let's denote our DIF framework without $\\hat{e}$ predictions (Figure 3 and remove the $g_\\phi$ MLP and $\\hat{e}$ branch) as DIF-Base.\n\n- **MAML**: Only Forecastor in the DIF-Base (with MAML optimization) since MAML does meta-learning at the optimization level.\n- **CoDA**: DIF-Base and set the dimension of $\\hat{z}_e=2$ (2 is the original paper setting).\n- **IRM**: DIF-Base. IRM regularization is at the loss level so no architecture change.\n- **VREx**: DIF-Base. VREx regularization is also at the loss level.\n\n**Inputs and Outputs**:\n\n- **MAML** takes X_p and outputs the forecasting trajectory directly.\n\n- **CoDA, IRM, VREx** take X_p as input and output function representations $\\hat{z}_c$ and $\\hat{z}_e$, then finally output the forcasting trajectory. The same as DIF.\n\n**Training**:\n\n- **MAML**: MAML optimization on Forcaster.\n- **CoDA, IRM, VREx** the same as DIF.\n\n**Inference**: (**Decoding X^c for the baselines**)\n\n- **MAML**: use the Forcaster with meta-parameters.\n- **CoDA, IRM, VREx**: the same as DIF: use the f_c branch.\n\n## Research Questions\n\n> At the start of section 5, the authors list their 6 research questions. I think the only 'real' research questions that this paper is addressing is 1 and 2, whereas the other ones are more 'additional evidence'. Maybe it is worth rewriting that to make the contribution of the paper more concise.\n\nWe agree with this valuable suggestion. In the revised paper, we will restructure Section 5 to emphasize Research Questions 1 and 2 as the primary contributions of our work.\n\nWe will reduce the remaining questions (3-6) to \"supplementary analyses\" that provide additional evidence for our claims rather than core research questions.\n\nWe thank the reviewer for the constructive feedback and hope the questions are addressed. We will incorporate all these improvements in the revised version of the paper.", "claims_and_evidence": "Broadly speaking, the main claims of the paper are that 1) the proposed method can learn invariant functions and that 2) learning invariant functions leads to better prediction performances. \n\nIn the experiments, 1 is demonstrated through qualitative examples where decoding f_c gives reasonable trajectories that correspond well to the underlying invariant function. For claim 2, the authors compare against invariant learning baselines and meta-learning baselines. In all cases, the proposed method seems to outperform the baselines across all datasets.\n\nMany extra results are included in the appendix which further corroborate their claims.", "methods_and_evaluation_criteria": "The problem setting considered in this paper is somewhat novel in that it requires similar ODEs from different environments (i.e. function forms). Prior works mainly work with environments where the only coefficient change across trajectories. As such, this paper modifies common dynamical systems such that they can have different functional forms across environments by adding extra terms such as damping. I believe this is sufficient for a proof-of-concept, although, as acknowledged by the authors in the limitations section, a more comprehensive benchmark for these environments would be helpful in evaluating the full potential of the approach. \n\nI have one question regarding the choice of datasets: It seems that across function environments, the changes are all additive (i.e. adding extra forcing terms to the ODE). Would we expect the method to still work if the changes are more non-linear? for example changing f(theta) = sin(theta) to f(theta) = sin(theta)^2.", "theoretical_claims": "I have checked the main theoretical claims (3.1, 3.2 and 3.3) and I am happy with their soundness.", "experimental_designs_or_analyses": "My main question regarding the experiments is about the baselines. It is unclear from the text how the baselines are implemented. While the proposed baselines are representative of the respective fields, it is not clear to me how these methods are adapted to the problem setting at hand: what are the architectures of the baselines? what are the input and outputs of these methods (do they also take in X_p and output z)? In the baselines, do you still have separate embeddings z_c and z_e?\n\nRelatedly, in Fig. 5b, how is X^c decoded for the baselines?\n\nSo far, the experimental setup seems sensible and qualitatively, the results for the approach look promising. But I think I can only assess the analyses and the results when the questions about the baselines are clarified.", "relation_to_broader_scientific_literature": "This paper brings concepts from invariant learning to the field of learning dynamical systems. I agree with the authors that applying the idea of invariances in this setting is non-trivial as it requires defining invariances in the function space. I believe the idea of learning invariant functions from observations is highly relevant to the dynamical system community, where the goal is not only to make forecast, but also to derive insight from the observed sequences. Here, the ability to learning meaningful and reusable functions from observations is particularly important.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "The work offers an original framework for learning reusable functions which is an important step towards ML methods that can discover generalisable and interpretable physics laws. The core concept is generally applicable and easy to understand.\n\nThe main weakness for me is that lack of clarity in terms of the experiment setup w.r.t. the baselines, which I think can be improved straight-forwardly. I would be happy to increase my score if this is clarified.", "other_comments_or_suggestions": "One minor comment: at the start of section 5, the authors list their 6 research questions. I think the only 'real' research questions that this paper is addressing is 1 and 2, whereas the other ones are more 'additional evidence'. Maybe it is worth rewriting that to make the contribution of the paper more concise.", "questions_for_authors": "See experimental design section and evaluation section.\n1. What would happen if changes across function environments are more drastic and non-linear?\n2. Please clarify the baselines.\n\n-------------\npost rebuttal:\nAfter the authors' clarification, I have raised my score to 4: accept.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel framework for extracting ODE-based physical laws from data using a new algorithm that disentangles invariant mechanisms based on structural causal models. This is an interesting new algorithm on an emerging area, and while experiments are on simple settings in general, they are convincing enough at this stage. Reviewers in general agree on the value of the paper, both in terms of technical novelty, rigor, and validation, although there are some concerns on the validity of the claims and whether they are clearly presented. What is more, all reviewers point on works from Brunton and Kranmer that could be discussed more. I agree on this. In fact, the paper should be compared with several related works published in the recent NeurIPS/ICML/ICLR [1, 2, 3, 4] in the related work section for the camera-ready version. This is not only for explaining the novelty better and having a more complete paper, but also giving more mass to the community.

[1] Pervez et al, Mechanistic neural networks for scientific machine learning, ICML 2024
[2] Chen et al., Scalable Mechanistic Neural Networks, ICLR 2025
[3] Liu et al., Amortized Equation Discovery in Hybrid Dynamical Systems, ICML 2024
[4] Auzina et al., Modulated Neural ODEs, NeurIPS 2023
Decision: Accept (poster)