Marrying Causal Representation Learning with Dynamical Systems for Science
Authors: Dingling Yao
Caroline Muller
Francesco Locatello
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal representation learning
Primary area: causal_inference

Abstract: Causal representation learning promises to extend causal models to hidden causal variables from raw entangled measurements. However, most progress has focused on proving identifiability results in different settings, and we are not aware of any successful real-world application. At the same time, the field of dynamical systems benefited from deep learning and scaled to countless applications but does not allow parameter identification. In this paper, we draw a clear connection between the two and their key assumptions, allowing us to apply identifiable methods developed in causal representation learning to dynamical systems. At the same time, we can leverage scalable differentiable solvers developed for differential equations to build models that are both identifiable and practical. Overall, we learn explicitly controllable models that isolate the trajectory-specific parameters for further downstream tasks such as out-of-distribution classification or treatment effect estimation. We experiment with a wind simulator with partially known factors of variation. We also apply the resulting model to real-world climate data and successfully answer downstream causal questions in line with existing literature on climate change.

Marrying Causal Representation Learning with Dynamical Systems for Science
Dingling Yao, Caroline Muller, and Francesco Locatello
Institute of Science and Technology Austria
Abstract
Causal representation learning promises to extend causal models to hidden causal variables from raw entangled measurements. However, most progress has focused proving identifiability results in different settings, and we are not aware of any successful real-world application. At the same time, the field of dynamical systems benefited from deep learning and scaled to countless applications but does not allow parameter identification. In this paper, we draw a clear connection between the two and their key assumptions, allowing to apply identifiable methods developed in causal representation learning to dynamical systems. At the same time, we can leverage scalable differentiable solvers developed for differential equations to build models that are both identifiable and practical. Overall, we learn explicitly control- lable models that isolate the trajectory-specine parameters for further downstream tasks such as experiment with a wind simulator with partially known factors of variation. We also apply the resulting model to worldworld climate data and successfully answer downstream causal questions in line with existing literature on climate change. Code is available at https://githubb.om www./oooo//aaaaaaaaaaaa SeatmiingA//rr--oe A/lerr/-dyaamiccctt tllyaammal-ssssemms...
classification or treatment effect estimation. We
1 Introduction
Causal representation learning (CRL) [54] focuses on provably retrieving high-level latent variables from low-level data. Recently, there have been many casual representation learning works compiling, in various settings, different theoretical identifiability results for these latent variables [8, 26, 32, 36, 37, 58, 60, 62, 65, 66, 70, 73]. The main open challenge that remains for this line of work is the broad applicability to real-world data. Following earlier works in disentangled representations (see [38] for summary of data sets), existing approaches have largely focused on visual data This challenging for various reasons. Most notably, it unclear what the causal variables should in computer vision problems and what would be interesting or relevant causal questions. The current standard is test algorithms on synthetic data sets with "made-up" latent causal graphs, e.g., with the object class of a rendered 3d shape causing its position, hue, and rotation [65].
In parallel, the field of machine learning for science [44, 49] shows promising results on various real-world time series data collected from some underlying dynamical systems. Some of these works primarily focus on time-series forecasting, i.e., building neural emulator that mimics the behavior of the given times series data [12, 13, 25]; while others try to additionally learn an explicit ordinary differential equation simultaneously [9. 10, 15, 18, 23, 56]. However, to the best of our knowledge, none of these methods provide explicit identifiability analysis indicating whether the discovered equation recovers the ground truth underlying governing process given time series observations; or even whether the learned representation relates to the underlying steering parameters. At the same time, many scientific questions are inherently causal, in the sense that physical laws govern the measurements of all the natural data we can record, e.g., across different environments and experimental settings. Identifying such underlying physical process can boost scientific understanding and reasoning in numerous fields; for example, in climate science, one could conduct
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sensitivity analysis of layer thickness parameter on atmosphere motion more efficiently, given neural emulator that identifies the layer thickness in its latent space. However, whether mechanistic models can be practically identified from data is so far unclear [54, Table 1].
This paper aims to identify the underlying invariant physical parameters from real-world time series, such as the previously mentioned layer thickness parameter, while still preserving the ability to forecast efficiently. Thus, we connect the two seemingly faraway communities, causal representation learning and machine learning for dynamical systems, by phrasing parameter estimation problems in
dynamical systems as a latent variable identification problem in CRL. The benefits are two folds: (1) we can import all identifiability theories for free from causal representation learning works, extending discovery methods with additional identifiability analysis and, e.g., multiview training constructs; (2)
showcase that the scalable mechanistic neural networks [47] recently developed for dynamical systems can be directly employed with causal representation learning, thus providing a scalable implementation for both identifying and forecasting real-world dynamical systems.
Starting by comparing the common assumptions in the field of parameter estimation in dynamical systems and causal representation learning, we carefully justify our proposal to translate any param- eter estimation problem into a latent variable identification problem: we differentiate three types of identifiability: full identifiability, partial identifiability and non-idenniffability We describe concrete scenarios in dynamical systems where each kind of identifiability can be theoretically guaranteed and restate exemplary identifiability theorems from the causal representation learning literature with slight adaptation towards the dynamical system setup. We provide a step-by-ste recipe for reformulating a parameter estimation problem into a causal representation learning problem and discuss the challenges and pitfalls in practice. Lastly, we successfully evaluate our parameter identification framework on various simulated and real-world climate data. We highlight the following contributions:
We establish the connection between causal representation learning and parameter estimation for differential equations by pinpointing the alignment of common assumptions between two communities and providing hands-on guidance on how to rephrase the parameter estimation problem as a latent variable identification problem in causal representation learning.
We equip discovery methods with provably identifiable parameter estimation approaches from the causal representation learning literature and their specific training constructs. This enables to maintain both the theoretical results from the latter and the scalability of the former.
We successfully apply causal representation learning approaches simulated and real-world climate data, demonstrating identifiability via domain- -speifc downstream causal tasks (OOD classification and treatment-effect estimation), pushing one step further the applicability of causal representation for real-world problems.
Remark on the novelty of the paper: Our main contribution is establishing connection between the dynamical systems and causal representation learning fields. As such, we do not introduce a new method per se. Meanwhile, this connection allows to introduce CRL training constructs in methods that otherwise would not have any identification guarantees. Further, it provides the first avenue for causal representation learning applications on real-world data. These are both major challenges in the respective communities, and we hope this paper will serve as building block for cross-pollination.
2 Parameter Estimation in Dynamical Systems
We consider dynamical systems in the form of
x(t) fo (x(t))
x x(0) xo, 0 ~ pa, t E [0, tmax]
(I)
where x(t) X S d denotes the state of system time t. fe E C² is some smooth differentiable vector field representing the constraints that define the system's evolution, characterized by set of physical parameters 0 E θ O X x ON where θ C R2N is an open, simply connected real space associated with the probability density pe. Formally, fo can be considered as a functional mapped from 0 through M O - C² (x'')) . In our setup, we consider time-invariant, trajectory-specift parameters 0 that remain constant for the whole time span [0. fmax], but variable for different trajectories. For instance, consider a robot arm interacting with multiple objects of different mass; a parameter O could be the object's masses m E R. in Newton's 's second law i(() m())m,, with F(L) denote the force applied at time t. Depending on the object the robot arm interacts with, m can take different values. following the prior distribution pg. x(0) = xo X

---

denotes the initial value of the system. Note that higher-order ordinary differential equations can always be rephrased as first-order ODE. For example, v-th order ODE in the following form:
x (0 (t) = = (a(t), x (1) (t), (t)...., ( (t),
can be written as xt) fo (x()), where x(t) = (a(t), xx1) ..),..... (.ll01111)))) E Rnd denotes state vector constructed by concatenating the derivatives. Formally, the solution of such dynamical system can be obtained by integrating the vector field over time: x(1) = So (x(()))))d
(t)...,,
What do we mean by "parameters"? The parameters that we consider can be both explicit and implicit. When the functional form of the ODE is given, like Newton's 's second law, the set of parameters defined explicitly and uniquely. For real-world physical processes where the functional form of the state evolution is unknown, such the sea-surface temperature change, we can consider latitude-related features as parameters. Overall, we use parameters to generally refer to any time-invariant, trajectory-specifee components of the underlying dynamical system.
Assumption 2.1 (Existence and uniqueness). For every Xy E X. O O there exists a unique continuous solution XØ [0, tmax] + X' satisfying the ODE (eq. (1)) for all t E [0, tmax] [22, 35].
Assumption 2.2 (Structural identifiability). An ODE (eq. (1)) is structurally identifiable in the sense that for any 0,, 0g E O xo, (t) = xo, (t) Vt E [0, |max] holds and only if 0 02 17, 67, 69).
Remark 2. Asm. 2.2 implies that it is in principle possible to identify the parameter 0 from a trajectory X9 [43]. Since this work focuses on providing concrete algorithms that guarantee parameter identifiability given infinite number of samples, the structural identifiability assumption is essential as a theoretical ground for further algorithmic analysis. It is noteworthy that a non-structurally identifiable system can become identifiable by reparamatization. For example, linear ODE x(() = abx(t) with parameters a, E R is structurally non- non-identifiable as a, are commutative. But if we define ab as the overall growth rate of the linear system, then e is structurally identifiable.
Problem setting. Given an observed trajectory x := (xo))).....x((tt)) E X over the discretized time grid T := (fo..... .T), our goal to investigate the identifiability of structurally identifiable parameters by formulating concrete conditions under which the parameter 0 is (i) fully identifiable, (ii) partially identifialle, or (iii) non-identifiable from the observational data. We establish the identifiability theory for dynamical systems by converting classical parameter estimation problems [7] into a latent variable identification problem in causal representation learning [54]. For both (i) and (ii), we empirically showcase that existing CRL algorithms with slight adaptation can successfully (partially) identify the underlying physical parameters.
3 Identifiability of Dynamical Systems
This section provides different types of theoretical statements on the identifiability of the underlying fime-invarint, vrejectory-specffee physical parameters 6 depending on whether the functional form of So is known or not. We show that the parameters from an ODE with a known functional form can be fully identified while parameters from unknown ODEs are in general non-identifible. However, by incorporating some weak form of supervision, such as multiple similar trajectories generated from certain overlapping parameters [16, 39, 65, 71], parameters from an unknown ODE can also be partially identified. Detailed proofs of the thcoretical statements are provided in App. B.
3.1 Identifiability of Dynamical Systems with Known Functional Form
We begin with the identifiability analysis of the physical parameters of an ODE with known functional form. Many real-world data we record are govemndd by known physical laws. For example, the bacteria growth in microbiology could be modeled with simple logistic equation under certain conditions, where the parameter of interest in this case would be the growth rate r E IR, and maximum capacity K E R4. Identifying such parameters would be helpful for downstream analysis. To this end, we introduce the definition of full identifiability of a physical parameter vector O.
Definition 3.1 (Full identifiability). A parameter vector 0 E θ is fully identified if the estimator i converges to the ground truth parameter 0 almost surely.
Definition 3.2 (ODE solver). An ODE solver F - X computes the solution x of the ODE fe = (0) (eq. (1)) over discrete time grid T (t1, (f1....,tr). tr).
3

---

Table Comparing typical assumptions of parameter estimation for dynamical systems and latent variable identification in causal representation learning, We justify that the common assumptions in both fields are aligned, providing theoretical ground for applying identifiable CRI. methods to learning-based based parameter estimation approaches in dynamical systems.
param. estimation
CRL assumption
Explanation
ref
assumption
ref
2.1
existence & uniqueness
determ. gen. 3.1
Both 2.1 and 3.1 implies determin- istic generative process.
supp(0) =
3.3
2.1 implies 3.3 as X9 uniquely ex-
ists for all 0 E O
2.2
structural identifibilityy
injectivity 3.2
2.2 implies 3.2 of the solution Xg.
Corollary 3.1 (Full identifiability with known functional form). Consider a trajectory x E X2T generated from a ODE (x()) satisfying Asms. 2.1 and 2.2, let Ô be an estimator minimizing the following objective:
[()) - ||(() x
then the parameter 0 fully-identified (Defn. 3.1) by the estimator O.
The estimator Ô of eq. (2) is considered some learnable parameters that can be directly optimized. If we have multiple trajectories x generated from different realizations of 0 ~ Pa. we can also amortize the prediction & using a smooth encoder X² - O. In this case, the loss above can be rewritten as: L(g) Ex, |II (x)()) x(t) then the optimal encoder g E argmin Lg))
Remark 3.1.
can generalize to unseen trajectories x that follow the same class of physical law f and fully identify
their trajectory-specific parameters 0.
Discussion. Many works on machine learning for dynamical system identification follow the principle presented in Cor. 3.1, and most of them solely differ concerning the architecture they choose for the ODE solver. For example, SINDy-like ODE discovery methods [9, 10, 23, 24, 47] approximate the ground truth vector field f using a linear weighted sum over set of library functions and learn the linear coefficients by sparse regression. For any ODE f that is linear in 0, i.e., the ground truth vector field is in the form of fo t) = Zim1 (x), for set of known base functions {Ør him()) SINDy-like approaches can fully identify the parameters by imposing some sparsity constraint. Another line of work, gradient matching [68], estimates the parameters probabilistically by modeling the vector field Se using a Gaussian Process (GP). The modeled solution x(1) is thus also a GP since GP is closed under integrals (a linear operator). Given the functional form of fe. the model aims to match the estimated gradient x and the evaluated vector field fe (x(()) by maximizing the likelihood, which is equivalent to minimizing the least-squares loss (eq. (2)) under Gaussianity assumptions. Hence, the gradient matching approaches can theoretically identify the underlying parameters under Cor. 3.1. Formal statements and proofs for both SINDy-like and gradient matching approaches are provided in App. B. Note that most ODE discovery approaches [9, 10, 23, 24, 47, 68] refrain from making identifiability statements and explicitly states it is unknown which settings yield identifiability.
m
3.2 Identifiability of Dynamical Systems without Known Functional Form
In traditional dynamical systems, identifiability analysis usually assumes the functional form of the ODE is known [43]; however, for most real-world time series data, the functional form of underlying physical laws remains uncovered. Machine learning-base bbased approaches for dynamical systems work in a blx manner and can clone the behavior of an unknown system [12, 13, 46), but understanding and identifiability guarantees of the learned parameters are so far missing. Since most of the physical processes are inherently steered by a few underlying fime-invaraint parameters, identifying these parameters can be helpful in answering downstream scientific questions. For example, identifying climate zone-related parameters from sea surface temperature data could improve understanding of climate change because the impact of climate change significantly differs in polar and tropical regions. Hence, we aim to provide identifiability analysis for the underlying parameters of an unknown dynamical system by converting the classical parameter estimation problem of dynamical systems into a latent variable identification problem in causal representation learning. We start by listing the common assumptions in CRL and comparing the ground assumptions between these two fields.
Assumption 3.1 (Determinism). The data generation process is deterministic in the sense that observation x is generated from some latent vector 0 using a deterministic solver F (Defn. 3.2).

---

Assumption 3.2 (Injectivity). For each observation x, there is only one corresponding latent vector 0, i.e., the ODE solve function F (Defn. 3.2) is injective in O.
Assumption 3.3 (Continuity and full support). Po is smooth and continuous on θ with > 0 a.e.
Assumption justification. Tab. summarizes common assumptions in traditional parameter estimation in dynamical systems and causal representation learning literature. We observe strong alignment between the ground assumptions in these two fields that justifies our idea of employing causal representation learning methods in parameter estimation problems for dynamical systems: (1) Asm. 2.1 implies that given fixed initial value Xo X, there exists unique solution x(t), E [0, tmax] for any fo with 0 E O. In other words, parameter domain O is fully supported (Asm. 3.3), and these ODE solving processes from F(0) (Defn. 3.2) deterministic, which aligns with the standard Asm. 3.1 in CRL. Since the ODE solution F(0) ( 2) is continuous by definition, the continuity assumption from CRL (Asm. 3.3) is also fulfilled. (2) Asm. 2.2 emphasizes that each trajectory x can only be uniquely generated from one parameter vector O which means the generating process F' (Defn. 3.2) injective in 0 (Asm. 3.2).
Next, we reformulate the parameter estimation problem in the language of causal representation learning. We first cast the generative process of the dynamical system fx(x(t) model by considering the underlying physical parameters 0 ~ Po as a set of latent variables. Given trajectory generated by set of underlying factors θ based the vector field fo (x()),, consider the observed trajectory as some unknown nonlinear mixing of the underlying 9 with the mixing process specified by individual vector field (x((()). This interpretation of observations aligns with the standard setup of causal representation learning; for instance, high-dimensional images are usually generated from some lower-dimensional latent generating factors through an unknown nonlinear process. Thus, estimating the parameters of unknown dynamical systems becomes equivalent to inferring the underlying generating factors in causal representation learning.
latent variable
After transforming the parameter estimation into a latent variable identification problem in CRL, can directly invoke the identifiability theory from the literature. Based Locatello al. [38, Theorem 1.], we conclude that the underlying parameters from an unknown system are in general non-identifiable. Nevertheless, several works proposed different weakly supervised learning strategies that can partially identify the latent variables [2, 8, 16, 39, 65, 71]. To this end, we define partial identifiability in the context of dynamical systems by slightly adapting the definition of Mloc-iideniiiiabiiityy proposed by Von Kügelgen et al. [65]:
Definition 3.3 (Partial identifiability). A partition 0g (0.)ss with S S [M] of parameter 0 O partially identified by an encoder g : X O if the estimator 0s := g(x)s contains all and only information about the ground truth partition 0s, i.e. Qg = h(0s) for some invertible mapping h: Os -> O where O == Xies O.
Note that the inferred partition 0g can be a set of entangled latent variables rather than a single one. In the multivariate case, one can consider the Os as a bijective mixture of the ground truth parameter Os.
Corollary 3.2 (Identifiability without known functional form). Assume dynamical system satisfying Asms 2.1 and 2.2, a pair of trajectories x, X generated from the same system f but specified by different parameters 0, 0, respectively. Assume a partition of parameters Og with S SC [N] is shared across the pair of parameters 0, Ô. Let Letg: : O be some smooth encoder and F: + + X" be some ieff-invertible smooth solver that minimizes the following objective:
Llg. F) = Ex.x (19())s - - [(((())
- xx1 +1 ((((x)) - -
(3)
Aligement
Siffieiency
then the shared partition 0g is partially identified (Defi. 3.3) by g in the statistical setting.
Discussion. We remark that an implicit ODE solver F is introduced in eq. (3) because the functional form fe is unknown. Intuitively, Cor. 3.2 provides partial identifiability results for the shared partition of parameters between two trajectories. We can consider the trajectories to be different simulation experiments but with certain sharing conditions, such as two wind simulations that share the same layer thickness parameter. This partial identifiability statement is mainly concluded from the theory in the multiview CRL literature [2, 8, 16, 39, 54, 65, 71J. Note that this corollary is one exemplary
5

---

Revonstruction
Input trajectory
Proj
Proj
Proj
Encoder
a Ci,lr bi,bi Si,t
ODE representation
Solver
Us C(or, 0) =
Test
Proj
Proj
Proj
Downstream causal analysis
Climate zone
Intervenc
Co Di,es S4,
Salver
0
Reconstruct colder seas
Figure 1: Model overview with sea surface temperature inputs: Our mechanistic identifier extracts the underlying time-invariant ttitude-relatee parameters 0, providing versatile neural emulator for downstream causal analysis.
demonstration of achieving partial identifiability in dynamical systems. Many identifiability results from the causal representation works can be reformulated similarly by replacing their decoder with differentiable ODE solver F. The high-level idea of multiview CRL is identify the shared part between different views by enforeing aligmment on the shared coordinates while preserving sufficient information representation. Alignment can be obtained by either minimizing the Lg loss between the encoding from different views on the shared coordinates [16, 65, 71] or maximizing the correlation on the shared dimensions correspondingly [41, 42]; Sufficiency of the learned representation is often prompted by maximizing the entropy [16, 65, 71, 74] or minimizing the reconstruction error [2, 8, 39, 54]. Other types of causal representation learning works will be further discussed in $ 5.
4 CRL-construct of Identifiable Neural Emulators for Dynamical Systems
This section provides a step-by-step construct a neural emulator that can (1) identify the time-inarrians, trajectory-specific physical parameters from some unknown dynamical systems the identifiability conditions are met and (2) efficiently forecast future time steps. Identifiability can be guarantced by employing causal representation learning approaches ($ 3) while forecasting ability can be obtained by using an efficient mechanistic solver [47] as a decoder. For the sake of simplicity, we term these identifiable neural emulators as identifiers. We remark that the general architecture remains consistent for most CRL approaches, while the learning object differs slightly in latent regularization, which specified by individual identifiability algorithms, Intuitively, the latent regularization can be interpreted as an additional constraint put the learned encodings imposed by the settin--specifi assumptions, such as alignment term in multiview CRL (Cor. 3.2). In the following, we demonstrate building an identifier in the multiview setting from scratch and showcase how can be easily generalized other CRL approaches with slight adaptation.
Architecture. Since the parameters of interest time-invarian and trajectory-specffe ($ 2), input the whole trajectory x = (x(t1)..... x(tr)) to a smooth encoder g X² - θ as shown in Fig. 1. Then, we decode the trajectory x from estimated parameter vector Ô : g(x) using mechanistic solver [47]. The high-level idea mechanistic neural networks is to approximate the underlying dynamical system using a set of explicit ODEs Ug C((x, Ô) = 0 with learnable coef- ficients a E &d, The explicit ODE family Ug can then be interpreted as a constrained optimization problem and can thus be solved using neural relaxed linear programming solver [47, Sec 3.1].
In more detail, the original design of MNN predicts the coefficients from the input trajectory x using an MNN encoder Imnn' however, as we enforee the estimated parameter 0 to preserve sufficient information of the entire trajectory x. we instead predict the coefficients a from the estimated parameter θ with the encoder Jmm θ Rda, Formally, the coefficients a are computed as 9mnn (0) where ê g(x). The resulting ODE family Ug provides a broad variability of ODE

---

parametrizations. A detailed formulation of Ug at [47, eq. (3)) is given by
E' 0) + ビド== @x (t, {u()1) = b(t: 0),
(4)
linear terms
noalinear terms
where (1)) is i-th order approximations of the ground truth state x. Like in any ODE solving in practice, solving eq. (4) requires discretization of the continuous coefficients in time (e.g.. (t: 0)). Discretizing the ODE representation Ug gives rise to:
Ci (0) C (1u, () bb s.l. (utz:
w,
where w denotes the initial state vector of the ODE representation Ug. To this end, we present the explicit definition of the learnable coefficients a := (ct.e. Ok bt, () with t E T,iE [1, E [r]. which is a concatenation of linear coefficients Cs.., nonlinear coefficients di,k. adaptive step sizes 8t and initial values w. Note that we dropped the the notation for simplicity, but all of these coefficients a are predicted from Ô, as described previously. At last, MNN converts ODE solving into a constrained optimization problem by representing the Ug using set of constraints, including ODE equation constraints, initial value constraints, and smoothness constraints [47, Sec 1.1. This optimization problem is then solved by neural relaxed linear programming solver [47, Sec 3.1] in time-parallel fashion, thus making the overall mechanistic solver scalable and GPU-friendly.
Learning objective and latent regularizers. Depending whether the functional form of the underlying dynamical system is known or not, the proposed neural emulator can be trained using the losses given in Cor. 3.1 or Cor. 3.2, respectively. When the functional form is unknown, employ CRL approaches to partially identify the physical parameters. We remark that the causal representation learning schemes mainly differ in the latent regularizers, specified by the assumptions and settings. Therefore, we provide a more extensive summary of different causal representation learning approaches and their corresponding latent regularizer in Tab. 6.
Related Work
Multi-environment CRL. Another important line of work in causal representation learning focuses on the multi- celliieenvrrommntt setup. where the data are collected from multiple different environments and thus non-identically distributed. Causal variable identifiability are shown under single node intervention per node with parametric assumptions on the mixing functions [3, 58, 62, 73] or on the latent causal model [11, 58). These parametric assumptions can be lifted by additionally assuming paired intervention per node, as demonstrated by [63, 66]. Overall, given the fruitful literature in multi-enviromment causal representation learning, we believe applying multi-environments methods to build identifiable neural emulators ($ 4) would be an exciting future avenue.
CRL and dynamical systems. Recent CRL works have been tackling the parameter identification problem in dynamical systems in a parametric setting. For instance, Rajendran et al. [50] considers Gaussian linear time invariant system with control input and Balsells-Rodas switching dynamical system. By contrast, we show identifiability in more general setting without specific parameteric assumptions on the dynamical systems and prove different granularity of parameter identification under different system prior system knowledge (full identifiability if parametric form known (Cor. 3.1) and partial identifiability when the system is unknown (Cor. 3.2)). Another closely related line of works, temporal causal representation learning, typically assume an "intervenable" time series the latent space, splitting the latent variables into two partitions, with one following the default dynamics and the other following the intervened dynamics. The goal of temporal CRL is to provably retrieve these time-varying latent causal variables, such as inferring the position of a ball from raw images over time [27, 30, 33, 36, 37, 72]. Unlike these temporal CRL works, our approach models dynamics directly in the observational space, focusing on the iime--mariiat,, mrajectory-specíff physical parameters such as gravity mass. Overall, framework addresses a different hierarchy of problems. We believe both problems are orthogonal yet equally important, encouraging cross poss-poolination in future work.
al. [5] assumes
ODE discovery. The ultimate goal of ODE discovery is to learn a human-interpretable equation for an unknown system, given discretized observations generated from this system. Recently, many machine learning frameworks have been used for ODE discovery, such as sparse linear regression [9, 10, 23, 53], symbolic regression [6. 15, 18], simulation-bast based inference [14, 56).
7

---

Becker et al. [6], d' A'Acooli et al. [18] exploit transformer-basee approaches to dynamical symbolic regression for univariate ODEs, which is extended by d' d'Ascoli al. [15] to multivariate case. Schröder and Macke [56] employs simulation-based variational inference to jointly learn the operators (like addition or multiplication) and the coefficients. However, this approach typically runs simulations inside the training loop, which could introduce a tremendous computational bottleneck when the simulator is inefficient. On the contrary, our approach works offline with pre-collected data, avoiding simulating on the fly. Although ODE discovery methods can provide symbolic equations for data from an unknown trajectory, the inferred equation does not have to align with the ground truth. In other words, theoretical identifiability guarantees for these methods are still missing.
Identifiability of dy dynamical systems. Identifiability of dynamical systems has been studied on a case-by-case basis in traditional system identification literature [4, 43, 64]. Liang and Wu [34] studied ODE identifiability under measurement error. Scholl et al. [55] investigated the identifiability of ODE discovery with non-parametric assumption, but only for univariate cases. More recently, several works have advanced in identifiability analysis of linear ODEs from a single trajectory [17, 48, 59]. Overall, current theoretical results cannot conclude whether an unknown nonlinear ODE can be identified from observational data. Hence, in our work, we do not aim to identify the whole equation of the dynamical systems but instead focus on identifying the time-invariant parameters.
6 Experiments
This section provides experiments and results on both simulated and real-world climate data. We first validate full parameter identifiability (Cor. 3. under a wide range of known dynamical systems, as demonstrated in 6.1. Next, consider in $$ 6.2 and 6.3 time series data govemndd by unknown physical process, so we employ the multiview CRL approach together with mechanistic neural networks build our identifiable neural emulator (termed as mechanistic identifier), following the steps in $ 4. We compare mechanistic identifier with three baselines: (1) Ada-GVAE [39], a traditional multiview model that uses vanilla decoder instead of mechanistic solver. (2) Time-invariant MNN, proposed by [47]. We choose this variant of MNN as our baseline for a fair comparison. (3) Contrastive identifier, contrastive loss-based CRL approach without a decoder [16, 65, 71]. We train mechanistic identifier using eq. (3) and other baselines following the steps given in the original papers. After training, we evaluate these methods on their identifiability and long-term forecasting capability.
6.1 Theory Validation: ODEBench
We demonstrate point-wise parameter identification results (presented RMSE, mean + std) over ODE system with known functional forms, including 63 dynamical systems from ODEBench [15] and the Cart-Pole system inspired by [72]. Results are summarized in Tab. 7. For each system, sample 100 tuples of the parameters A within a valid range to e.g., preserve the chaotic properties. For each tuple, we solve the problem outlined in Cor. 3.1, 1, by either regressing the estimated observation F(O) onto the true observation x. or equivalently, regressing the estimated vector field fo(x onto the derivatives x, given the same initial conditions. Note the data derivatives x can be obtained from numerical approximation in case it is not given priori. The resulting root-mean-squee (RMSE) is calculated and averaged across the parameter dimensions. We report the mean and standard deviation of these averaged RMSES over the 100 independent runs. From Tab. 7, we observe highly accurate point estimates for all stationary system parameters 6, thereby validating Cor. 3.1 across various experimental settings.
deviation
6.2 Wind Simulation
Experimental setup. Our experiment considers longitudinal and latitudinal wind velocities (also termed u, e wind components) from the global wind simulation data generated by various layer-thickness parameters. Fig. 2 depicts the wind simulation output at certain time point. To train the multiview approaches, we generate tuple of three views: After sampling the first view x" randomly throughout the whole training set, we sample another trajectory x² from a different location which shares the same simulation condition as the first one, compared to the first view, the third view x3 is then sampled from another simulation but at the same location. Overall, x², x2 share the global simulation conditions like the layer thickness parameter
Figure 2: Wind simulation: u, components [m/s] of simulated air motion over the globe.
8

---

10
Ads-VVAE
6/6
Table 2: Performance evaluation on the SST- V2 data on various types of tasks. Results averaged over three ran-
TLANX
0.6
6.6
06
dom seeds with standard deviation, provided as (m + std).
Coner. Ml.
1.0
0.5
0.
SST V2
-05
Acc:(ID(t)
Acc.(ODD((1)
Forcenst. crror())
Mrrh. M..
65
Ada -VAE
0.468 #: 0.001
0.467 + 0.000
0.043 + 0.044
Figure 3: Prediction accuracy layer thickness parameter on wind simulation data, evaluated on individ- ual encoding partitions S1, S2, S3. Re- sults averaged from three random runs.
TI-MNN
0.697 d: 0.049
0.668 :: 0.074
0.024 +: 0.018
Contr. Identifier
0.904 0.011
0.861
0.022
Mech. Identifier
1.90222 0.005
0.821 + 0.016
0.007 0.003
while x', x² only share the local features. All three views share global atmosphere-related features that are not specified as simulation conditions. More details about the data generation process and training pipeline are provided in App. C.2.
Parameter identification. In this experiment, we use the learned representation to classify the ground-truth labels generated by discretizing the generating factor layer thickness, and report the accuracy in Fig. 3. In more detail, we use latent dim=12 for all models and split the learned en- codings into three partitions S2, Sg, with four dimensions each. Then, we individually predict the ground truth layer thickness labels from each partition. According to the previously mentioned view- generating process, the layer thickness parameter should be encoded in S₁ for both contrastive and mechanistic identifiers. This hypothesis is verified by Fig. 3 since both contrastive and mechanistic identifiers show high accuracy of acc=1 I in the first partition S₁ and low accuracy in other partitions. On the contrary, Ada-GVAE and 7I-MNN performed significantly worse with an average acc. of 60% everywhere. Overall, Fig. shows both the necessity of explicit time modeling using MNN solver (compared to Ada-GVAE) and identifiability power of multiview CRL (compared to T1-MNN).
6.3 weal-world Sea Surface Temperature
Experimental setup. We evaluate the models on sea surface temperature dataset SST-V2 [21]. For the multiview training, we generate a pair trajectories from small neighbor region (+5%) along the same latitude. We believe these pairs share certain climate properties as the locations from the same latitude share roughly the amount of direct sunlight which will directly affect the sea surface temperature. Further infromation about the dataset and training procedure is provided in App. C.2.
Time series forecasting. We chunk the time series into slices of 4 years in training while keeping last four years as out-of-distribution forecasting task. To predict the last chunk, we input data from 2015 to 2018 to get the learned representation O. Since we assume Ô to be time-inavriant, we decode i together with 10 initial steps of 2019 to predict the last chunk. Note that contrastive identifier is excluded from this task as it does not have a decoder. As shown in Tab. 2, the forecasting performance of mechanistic Identifier surpasses Ada-GVAE by a great margin, showcasing the superiority of integrating scalable mechanistic solvers in real-world time series datasets. At the same time, TI-MNN performed worse and unstably despite the MNN component, verifying the need of the additional information bottleneck (parameter encoder g) and the multiview learning scheme.
Climate-zone classification. Since there is no ground truth latitude-related parameters available, we design downstream classification task that verifies our learned representation encodes the latitude- related information. The goal of the task is to predict the climate zone (tropical, temperate, polar) from the learned shared representation because the latitude uniquely defines climate zones. We evaluated the methods in both in-distribution (ID) and out-of- distribbbtttoon (OOD) setup for all baselines. In the OOD setting, we input data from longitude 10e longitude 360 when training the classifier while keeping the first 10 degree as our nuttof-distribution test data. Tab. 2 show that both contrastive and mechanistic identifiers perform decently, supporting the applicability of identifiable multiview CRL algorithms in dynamical systems. Overall, the performance of multiview CRL-based approaches (contrastive and mechanistic identifiers) far exceeds Ada-GVAE and T7-MNN, again showcasing the superiority of the combination of causal representation learning and mechanistic solvers.

---

non-identified identified
Estimated ATE trend
4x
3x
22
θ
T
X
1990
1995
2000
2005 Time
2010 2015 2020
Figure 4: Left: Underlying causal model for SST-V2 data, 0: covariates (latitude-relatdd parameters of interest), X: outcome (zonal average temperature), T: treatment (tropical T or polar T 1). Right: Comparison on ATE change ratio between identified and non-identified parameters, computed by AT'agear)- E((199))) A(199) AT E(1900) averaged over three runs.
Average treatment effect estimation. We further investigate the effect of climate zone on average temperature along specific latitude through average treatment effect (ATE) estimation. Formally, we consider the latitudinal average temperature as outcome Y. two climate zones (tropical (T = 0). polar(T = 1)) binary treatments, and the predicted latitude-specific features unobserved mediators. Formally, ATE is defined as: ATE := (E = 1)] do = 0)]. Since ATE cannot be computed directly [20], estimate it using the popular AIPW estimator [52]. Fig. 4 illustrates that the estimated ATE from the non-identified representation lacks discernible pattern [51] whereas the identified representation exhibits a noisy yet clear increasing trend, indicating the global warming effect. This is because the non-identified representation failed to isolate the covariates leading to biased treatment effect estimates. To estimate treatment effects, the covariates (i.e., the latitude-related parameters we identify) must not be influenced by the treatment (i.e., the climate zones). Otherwise, they become confounders, leading to incorrect estimates [19].
7 Limitations and Conclusion
In this paper, we build bridge between causal representation learning and dynamical system identification. By virtue of this connection, we successfully equipped existing mechanistic models (focusing on [47] practice for scalability reasons) with identification guarantees. Our analysis covers large number of papers, including [9, 10, 23, 24, 47, 68] explicitly refraining from making identifiability statements. At the same time, our work demonstrated that causal representation learning training constructs are ready to be applied in the real world, and the connection with dynamical systems offers untapped potential due to its relevance in the sciences. This was an overwhelmingly acknowledged limitation of the causal representation learning field [3, 11, 16, 39, 58, 62, 65, 71]. Having clearly demonstrated the mutual benefit of this connection, we hope that future work will scale up identifiable mechanistic models and apply them scientific questions. Nevertheless, this paper has several technical limitations that could be addressed in future work. First of all, the proposed theory explicitly requires determinism one of the key assumptions (Asm. 3),,, which directly excludes another important type of differential equation: Stochastic Differential Equations. Second, assume directly observe the state without considering measurement noise. Although the empirical results were promising on real-world noisy data ($ 6.3), we believe explicitly modeling measurement noise would elevate the theory. Finally, our identifiability analysis focuses on the infinite data regime, which is unrealistic in real-world scenarios.
more complex dynamical systems and real
Acknowledgments
We thank Niklas Boers for recommending the Speedy Weather simulator and Valentino Maiorca for guidance on Fourier transformation for SST data. We are also grateful to Shimeng Huang and Riccardo Cadei for their feedback on the treatment effect estimation experiment and to Jiale Chen and Adeel Pervez for their assistance with the solver implementation. Finally, we appreciate the anonymous reviewers for their insightful suggestions, which helped improve the manuscript.
10

---

References
[1] Nasir Ahmed, I_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90-93, 1974. 18
[2] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-
15528, 2022. 5, 6, 22
[3] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal repre- sentation learning. International Conference on Machine Learning, pages 372-407. PMLR,
2023. 7, 10
[4] Karl Johan Áström and Peter Eykhoff. System identification-a survey. Automatica, 7(2):
123-162, 1971. 8
[5] Carles Balsells-Rodas, Yixin Wang, and Yingzhen Li. On the identifiability of switching dynamical systems, 2023. 7
[6] Sören Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus. Predicting ordinary differential equations with transformers. In International Conference on Machine Learning, pages 1978-2002. PMLR, 2023. 7,8
[7] Ror Bellman and Karl Johan Åström. On structural identifiability. Mathematical biosciences, 7 (3-4):3293339 1970. 3
[8] Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022. 5,
[9] Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-33) -937, 2016. 1, 4. 7. 10, 17
[10] Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Sparse identification of nonlinear dynamics with control (sindyc). Puper-pnprsOnL..ne, 49(18):710-715, 2016. 1, 1,4, 7, 10, 17
[11] Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Advances in Neural Information Processing Systems, 36, 2024. 7, 10
[12] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018. 4, 20
[13] Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordinary differential equations. International Conference on Learning Representations, 2021. 1,
4, 20
[14] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-basee based inference. Proceedings of the National Academy of Sciences, 1 117(48):30055-3000 2020.
[15] Stéphanc d' Ascoli, Sören Becker, Philippe Schwaller, Alexander Mathis, and Niki Kilbertus. ODEFormer: Symbolic regression of dynamical systems with transformers. In International Conference Learning Representations, 2024. 7, 8, 23
[16] Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning, arXiv preprint arXiv:2303.09 1666, 2023. 3, 5, 6, 8, 10, 17, 22
[17] Xiaoyu Duan, JE Rubin, and David Swigon. Identification of affine dynamical systems from single trajectory. Inverse Problems, 36(8):085004, 2020. 8
[18] Stéphane d' Ascoli, Pierre-Alexandrr Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In International Conference on Machine Leurning, pages 4520-4536. PMLR, 2022. 7,8
[19] Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S Kohane, and Mihacla van der Schaar. Causal

---

machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958-99) 968, 2024.
10
[20] Paul W Holland. Statistics and causal inference. Journal of the American statistical Association,
8 11396))945-960, 1986. 10
[21] Boyin Huang, Chunying Liu, Viva Banzon, Eric Freeman, Garrett Graham, Bill Hankins, Tom Smith, and Huai-Min Zhang. Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1. Journal of Climate, 34(8):2923- 2021. 9, 19
[22] Edward L. Ince. Ordinary differential equations. Courier Corporation, 1956.
[23] Kadierdan Kaheman, Nathan Kutz, and Steven L Brunton. Sindy-pi: robust algorithm for
parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society
A, 144766222222222022779 2020. 1, 4, 10
[24] Alan Kaptanoglu, Jared L Callaham, Aleksandr Aravkin, Christopher Hansen, and Steven Brunton, Promoting global stability in data-driven models of quadratic nonlinear dynamics.
Physical Review Fluids, 699):094 2021. 4, 10
[25] Patrick Kidger, Ricky Q. Chen, and Terry Lyons. "hey, that's not an ode": Faster ode adjoints via seminorms. International Conference on Machine Learning, 2021. 1,
[26] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15687-15701, Curran Associates, Inc., 2022.
[27] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007,109330 2020.
[28] Milan Klöwer and the Speedy Weather.ji Contributors. Speedyweather jl. 2023. URL https://gi http://ggtttuuu..c htttpp///ggthtbb...cm//eeeeaaete hwtm///peetyaaaaaeeeeeeeeeaaaaaa jj. 18
[29] Sébastien Lachapelle and Simon Lacoste-Julien. Partial disentanglement via mechanism sparsity.
arXiv preprint arXiv:2207.07773,, 2022. 21, 22
[30] Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pages 428 484. PMLR, 2022.
[31] Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagka,, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning. In International Conference on Machine Learning, pages 18171-18206 PMLR, 2023. 21, 22
[32] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-produe extrapolation. Advances in Neural Information Processing Systems, 36, 2024. 1
[33] Zijian Li, Ruichu Cai, Zhenhui Yang, Haiqin Huang, Guangyi Chen, Yifan Shen, Zhengming Chen, Xiangchen Song, Zhifeng Hao, and Kun Zhang. When and how: Learning identifiable latent states for nonstationary time series forecasting. arXiv preprint arXiv:2402.12761 2024. 7, 22
[34] Hua Liang and Hulin Wu. Parameter estimation for differential equation models using framework of measurement error regression models. Journal of the American Statistical Association, 103(484):1570-1583 2008. 2008.
8
[35] Ernest Lindelöf. Sur I' l'application de la méthode des approximations successives aux équations différentielles ordinaires du premier ordre. Comptes rendus hebdomadaires des séances de I' l'Académie des sciences, 116(3):454-477, 1894.
12

---

[36] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects interactive systems. In The Fleventh International Conference on Learning Representations, 2022. 1,7
[37] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 25577-36033 PMLR, 2022. 7,7
[38] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 41 14
4124, PMLR, 2019. 1, 2,
[39] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference Machine Learning, pages 6348 6359. PMLR, 2020. 3, 5, 6, 8, 10, 18, 22
[40] Peter Y Lu, Joan Ariño Bernad, and Marin Soljačić. Discovering sparse interpretable dynamics from partial observations. Communications Physics, 5(1):206, 2022. 17, 22
[41] Qi Lyu and Xiao Fu. On finite-sample identifiability of contrastive learning- based nonlinear independent component analysis. In International Conference on Machine Learning, pages
14582- 14600. PMLR, 2022. 6
[42] Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. Understanding latent correlation- based multiview learning and self-supervision: An identifiability perspective. arXiv preprint arXiv:2106.07/15
2021. 6
[43] Hongyu Miao, Xiaohua Xia, Alan S Perelson, and Hulin Wu. On identifiability of nonlinear ode models and applications in viral dynamics. SIAM review, 53(1):3-39, 2011. 3, 4, 8
[44] Eric Mjolsness and Dennis DeCoste. Machine learning for science: state of the art and future prospects. science, 293(5537):2511-2555 2001.
[45] Gemma Elyse Moran, Dhanya Sridhar, Yixin Wang, and David Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research, 2022. ISSN 2835- 8856. URL https: /ttp///eeeeeeeeeeeeeeeeeeeeeeeee net/ orrumiid=vdoonnwZb... 21, 22
[46] Alexander Noreliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour augmented neural odes. Advances in neural information processing systems, 33:5911-5921 2020. 4
[47] Adeel Pervez, Francesco Locatello, and Efstratios Gavves. Mechanistic neural networks for scientific machine learning. International Conference on Machine Learning, 2024. 2, 4, 6, 7, 8, 10, 18, 20
[48] Xing Qiu, Tao Xu, Babak Soltanalizadeh, and Hulin Wu. Identifiability analysis of linear ordinary differential equation systems with single trajectory. Applied Mathematics and Computation, 430: 127260, 2022. 8
[49] Maithra Raghu and Eric Schmidt. A survey of deep learning for scientific discovery. arXiv preprint 2rXiv:2003.11753 2020. 1
[50] Goutham Rajendran, Patrik Reizinger, Wieland Brendel, and Pradeep Kumar Ravikumar, An interventional perspective identifiability in gaussian Iti systems with independent component analysis. In Causal Learning and Reasoning, pages 41-70. PMLR, 2024. 7
[51] Mika Rantanen, Alexey Yu Karpechko, Antti Lipponen, Kalle Nordling, Otto Hyvärinen, Kimmo Ruosteenoja, Timo Vihma, and Ari Laaksonen. The arctic has warmed nearly four times faster than the globe since 1979. Communications earth environment, 3(1):168, 2022. 10
[52] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association 89(427):846-866 866. 1994. 10
13

---

[53] Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. data-driven discovery of partial differential equations. Science advances, 364)xe1602614 2017. 7
[54] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning, Proceedings of the IEEE, 109(5)611-6334, 2021. 1, 2, 3, 5,
[55] Philipp Scholl, Aras Bacho, Holger Boche, and Gitta Kutyniok. The uniqueness problem of physical law learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. 8
[56] Cornelius Schröder and Jakob Macke. Simultaneous identification of models and parameters
of scientific simulators. arXiv preprint 2rxxv:2305.55174 2023. 7, 8
[57] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. Advances in Neural Information Processing Systems, 36, 2024. 22
[58] Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentangle- ment via interventions. In International Conference on Machine Learning, volume 202, pages 32540-32560. PMLR, 2023. 1, 10
[59] Shelby Stanhope, Jonathan E Rubin, and David Swigon. Identifiability of linear and linear-in- parameters dynamical systems from a single trajectory. SIAM Journal on Applied Dynamical Systems, 1 1794)11792-1815, 2014.
[60] Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domall causal representation learning. Advances in Neural Information Processing Systems, 36, 2024.
[61] Francesco Tonolini, Bjøm Sand Jensen, and Roderick Murray-Smite Variational sparse coding. In Uncertainty in Artificial Intelligence, pages 690-700. PMLR, 2020. 22
[62] Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Score- based causal representation learning from interventions: Nonparametric identifiabil- ity. In Causal Representation Learning Workshop at NeurIPS 2023, 2023. URL
1, 7, 10
[63] Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning, In International Conference on Artificial Intelligence and Statistics, pages 2314-2322. PMLR, 2024. 7
[64] Alejandro F Villaverde, Antonio Barreiro, and Antonis Papachristodoulou. Structural identifi- ability of dynamic systems biology models. PLoS computational biology, 12(10)ee0055133 2016. 8
[65] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:
16451-16467, 2021. 1,3,5, 5, 6, 8, 10, 17, 21,11,2
[66] Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024. 1,7
[67] Eric Walter, Lue Pronzato, and John Norton. Identification of parametric models from experi- mental data, volume 1. Springer, 1997. 3
[68] Philippe Wenk, Alkis Gotovos, Stefan Bauer, Nico S Gorbach, Andreas Krause, and Joachim M Buhmann. Fast gaussian process based gradient matching for parameter identification in systems of nonlinear odes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1351-1360. PMLR, 2019. 4, 10, 17
[69] Franz- Georg Wieland, Adrian L. Hauber, Marcus Rosenblatt, Christian Tönsing, and Jens Timmer. On structural and practical identifiability. Current Opinion in Systems Biology, 25: 60-69, 2021. 3
14

---

[70] Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. International Conference on Machine Learning, 2024. 1, 21, 22
[71] Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. In The Twelfth International Conference on Learning Representations, 2024, URL https: 10, 17, 18, 21, 22
3, 5, 6,
[72] Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled represeettionnnnnnnnn learning. Advances in Neural Information Processing Systems, 11-26492-265033 2022. 7, 22, 23
[73] Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shan- mugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances Neural Information Processing Systems, 36, 2024. 1,
[74] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990, PMLR, 2021. 6
15

---

Appendix
Table of Contents
A Notation and Terminology
16
B Proofs
16
B.1 Proofs for full identifiability
B.2 Proofs for partial identifiability
17
C Experimental results
18
C.I Wind simulation: SpeedyWeather.3 .jl
18
C.2 Sea surface temperature: SST-V2
19
C.3 Experiments and computational resources.
Discussion
A Notation and Terminology
Vector field
N
Dimensionality of parameter 0
Parameter domain
Time-invariant parameters for ODE f
M
Function that maps from 0 to fe
T
Number of time steps
max
End of the time span Discretized time grid of size
T
x(t)
ODE solution at time t
ODE solution for the time grid T
X
State-space domain
XT
Trajectory domain over time grid T
B Proofs
B.1 Proofs for full identifiability
Corollary 3.1 (Full identifiability with known functional form). Consider a trajectory x E T generated from a ODE (x(()) satisfying Asms. 2.1 and 2.2, let θ be an estimator minimizing the following objective:
c(θ) = F(0) 1
(2)
then the parameter is fully-identifie (Defn. 3.1) )) by the estimator Ô.
Proof. We begin by showing the global minimum of L()) exists and equals zero. Then, we show by contradiction that any estimators 0 that obtains this global minimum has to equal the ground truth parameters 0.
Step 1. We show that the global minimum parameter O E O, then by definition of the ODE solver F (Defn. 3.2), we have:
be obtained for L(0). Consider the ground truth
(( [[(())
x
= ||x x|2 0.
(6)
16

---

Step 2. Suppose for a contraction that there exists 0 θ that minimizes the loss eq. (2) but differs from the ground truth parameters 0, i.e., 0* + 0. This implies:
(( = - [((0)) - 2 =0
Note that L()") can be rewritten as:
T [( Σ 11 x((4)112
(8)
To make sure the sum is zero, each individual term has to be zero, that is F(O = x(tx), Vt
(4,...., (T). According to the uniqueness assumption of the ODE (Asm. 2.1), this implies 0" 0, which leads to contradiction.
Thus, we have shown that minimizing eq. (2) will yield the ground truth parameter 0. In other words, any estimator O that minimizes eq. (2) fully identifies 0.
Full identifiability with closed form solution when fe is linear in 0. We show that a closed-form solution can be obtained through linear least squares when the vector field fe linear in 0 and if we observe a first-order trajectory. A first-order trajectory means the first-order derivatives are included in the state-space vector. This statement is formalized as follows:
Observation B.1. Given a first-order trajectory (x, x) (x(t), x(())EET generated from a dynamical system fo(x (x)) satisfying Asms. 2.1 and 2.2. In particular, this ODE Jo can be written weighted sum of a set of base functions {Øi...., 6m i.e., fo is linear in O:
fe(x(t)) m (x).
(9)
Define 4x [xx(((()])e(())ee))) E R then the global optimum of the loss eq. (2) is given by
0 ( OxX
(10)
As a identify the underlying physical parameters O even with a closed-form solution if the underlying vector field fe is can be represented as sparse weighted sum of the given base functions {6.
direct implication, SINDY-like approaches [9, 10, 40] and gradient matching [68] can fully
B.2 Proofs for partial identifiability
Corollary 3.2 (Identifiability without known functional form). Assume a dynamical system f satisfying Asms. 2.1 and 2.2, pair of trajectories X generated from the same system f but specified by different parameters 0, Ô, respectively. Assume partition of parameters Og with S SC [N] is shared across the pair of parameters 0, O. Let g : X - θ be some smooth encoder and :: O X be some left left-invertible smooth solver that minimizes the following objective:
L(g. F) = Ex,x (l((x)s - g(x)s112 [f((()) -(oxx))xxlxx x1, + f(o(x)) 2
Aligament
Sifficiency
then the shared partition Og is partially identified (Defi. 3.3) by in the statistical setting.
Proof. This proof can be directly adapted from the proofs with by Daunhawer et al. [16], Von Küggll- gen et al. [65], Yao al. [71] with slight modification. So we briefly summarize the Step 1. and Step 2. that are imported from previous work and focus on the modification (Step 3.).
Step 1. We show that the loss function eq. (3) is lower bounded by zero and construct optimal encoder
g" : X + O that reach this lower bound. Define g" : XxT + Q = F-1 as the inverse of the ground truth data generating process, i.e., for all trajectories F(0) that generated from parameter 0, it holds:
(x)
(11)
Thus, we have shown that the global minimum zero exists and can be obtained by the inverse mixing function F-1, x² + ② (Defn. 3.2).
17

---

Step 2. We show that any optimal encoders 9 that minimizes eq. (3) must have the alignment equal zero, in other words, it has to satisfy the invariance condition, which is formalized as
g(x)s = g(x) 0.8.
(12)
Following Yao et al. [71, Lemma D.3], we conclude that both g(x) and g(x)s can only depend on information about the shared partition about the ground truth parameter 0ş. In other words,
g(x)s g())s h(Os)
(13)
for some smooth h: Os - Os.
Step 3. At last, show that h is invertible. Note that any optimal encoders g that minimizes eq. (3) must have zero reconstruction error on both x and x. Taking x as an example, we have
E F(g(x)) - x x
(14)
which implies
F(o(x)) x
0.S.
(15)
If two continuous functions F(9(x)) and x equals almost everywhere on θ then they are equal everywhere on O, which implies:
F(x(x)) x
4A E +
(16)
Substituting x with the ground truth generating process F:
F(g(x)) = F()) 60 E θ
(17)
applying the left inverse of F. we have:
p-1 o F(((x) = = o F(0) YO E E
(18)
i.e.
g(x) = f-1 F(0) F o F F(s.. 0s)
4O O
(19)
Define h² f- F note that h" is bijective as composition of bijections. Imposing the invariance constraint, we have 9xx)s = ht (0s, 0g)s. Since (x)s cannot depend on 0g as shown in Step 2, we have g(x)s (0s) with == hīs Og + Og.
Thus we have shown that g(x)s partially identifies 0g.
C Experimental results
General remarks. All models used in the experiments ($ 6) (Ada-GVAE, GVAE. TI-MNN, contrastive identifier, mechanistic identifier) were built upon open-sourced code provided by the original works [39, 47, 71], under the MIT license. For mechanistic identifters, we add a regularizer multiplier on the alignment constraint (Defn. 3.3), which is shown in Tabs. 3 and 5.
C.1 Wind simulation: SpeedyVeather.1 jjl
We simulate global air motion using using the Shall lalloowaateMMdee from speedy weather Julia package [28]. We consider layer thickness as the primary generating factor in ShallowWaterNodel varying from 8e][m] to 2e4[m], which is reasonable range given by the climate science literature. Taking the minimal and maximal values, we simulate the wind in a binary fashion and obtain 9024 trajectories across the globe under different conditions. Each trajectory constitutes three output variables discretized on ts=121 time steps, on a 3D resolution grid of size: latitude 1at=47; longitude 1on=96; level 1ev=1. The three output variables represent wind component (parallel to longitude), wind component (parallel to latitude), and relative vorticity, respectively. An illustrative example of all three components is depicted in Fig. 5. further details about the simulation output are provided in Tab, 4. In particular, to train more efficiently, we pre-process the data using a discrete cosine transorrm (DCT) proposed by Ahmed et al. [1] and only keep the first 50% frequencies. This is feasible as the original data possesses a certain periodic pattern, as shown in Fig. 6.
For all baselines, we train the model till convergence. More training and test details for the tasks in 6.2 are summarized in Tab. 3. To validate identifiability, we use cogisticRegressio.. model from scikit- ceikittleaarn learn in its default setting to evaluate the classification accuracy in Fig. 3.
18

---

120%
DR0TE
120"W
120E
220TW
220
20"5
2015
20'5
20'5
2015
2015
RDVE
GOF
60'F
Figure Example of wind simulation: Left: longitudinal wind velocity (u) [m/s]. Middle: latitudinal wind velocity (m)[m/s), Right: relative vorticity (vor) [1/s].
Table 3: Training setup for wind simulation in $ 6.2. Non-ap con-applicable fields are marked with X.
Ada-GVAE
TI-MNN
Cont. Identifier
Mech. Identifier
Pre-process
DCT
DCT
DCT
DCT
Encoder
6-layer MLP
6-layer MLP
6-layer MLP
6-layer MLP
Decoder
6-layer MLP
6-layer MLP
X
3 proj. x 6-layer MLP
Time dim
121
121
121
121
State dim
2
2
2
2
Hidden dim
1024
1024
1024
1024
Latent dim
12
12
12
12
Optimizer
Adam
Adam
Adam
Adam
Adam: learning rate
le-5
1e-5
le-5
le-5
Adam: betal
0.9
0.9
0.9
0.9
Adam: beta2
0.999
0.999
0.999
0.999
Adam: epsilon
le-8
le-8
1e-8
le-8
Batch size
1 1128
1128
1128
1128
Temperature 7
X
0.1
X
Alignment reg.
X
X
10
# Initial values
10
10
X
10
# Iterations
< 30,000
30,000
< 30,000
<30,00
# Seeds
3
3
3
C.2 Sea surface temperature: SST-V2
The sea surface temperature data SST- V2 [21] contains the weekly sea surface temperature data from 1990 to 2023, on a resolution grid of 180 360 (latitudes x longitudes). An example input depicted in Fig. 8. Each time series contains 1727 times steps. To generate multiple views that share specific climate properties, we sample two different trajectories from small neighbor region (+:5) along the same latitude, the latitude differs in the amount of direct sunlight thus directly affecting the sea surface temperature.
For a fair comparison, we train all baselines till convergence following the setup summarized Tab. 5. Similar to the wind simulation data, we pre-process SST-V2 data using DCT and keep the first 25% frequencies, as the latitude-related parameters of interest primarily influence long-term dependencies, such as seasonality, which are predominantly captured by low-frequency components. Fig. 7 shows example of predicted trajectories over three randomly sampled locations. As for the downstream classification task, we use LogisticRegresse model from scikit-lears in its default setting to evaluate the classification accuracy in Tab. 2.
C33 Experiments and computational resources
In this paper, we train four different models, each over three independent seeds. All 12 jobs ran with 24GB of RAM. 8 CPU cores, and single node GPU, which is, most cases, NVIDIA GeForce RTX2080Ti. Given different model sizes and convergence rates, the required amount of compute
19

---

Table 4: Wind simulation: output variables.
Output variable [unit]
Shape
Longitudinal wind velocity (u) [m/s]
(ts, lev, lat, lon)
Latitudinal wind velocity (v) [m/s] Relative vorticity (vor) [1/s]
(ts, lev, lat, lon)
(ts, 1ev, lat, lon)
1oc
comeoonees pompooret
ground trudis
20000
25
50
75
100
125
0
25
30
75
100
125
25
50
75
100
125
Figure 6: Wind simulation: mechanistic identifier reconstruction highly irregular time series. The first half of the trajectory is provided as initial values, while the second half is predicted.
could vary slightly, despite the pre-fixed training epochs. Thus, report upper bound of the compute hours on NVIDIA GeForce RTX2080Ti. On average, all runs converge within 22 GPU hours. Therefore, the experimental results in this paper can be reproduced with 264 GPU hours.
D Discussion
Why mechanistic neural networks [47]. As mentioned $ 4, the ODE solver F given in Cors. 3.1 and 3.2 can be interpreted as the decoder traditional representation learning regime; however, sev- eral challenges arise when integrating ODE solving in the training loop: First of all, the ODE solver must be differentiable to utilize the automatic differentiation implementation of the state-of-the-an deep learning frameworks; this obstacle has been tacked by the line of work termed NewralODE. which models the ODE vector field using neural network thus enable differentiability [12, 13, 25]. Nevertheless, most differentiable ODE solvers solve the ODE autoregressively and thus cannot be par- allelized by the GPU very efficiently. Dealing with long-term trajectories (for example, weekly climate data during the last few decades) would be extremely computationally heavy. Therefore, we advocate for time- and memory-
differentiable ODE solver: the mechanistic neural networks [47].
1
loc
0
loc 2
1
0
loc 3
1
ground truth
recon
1990
1995
2000
2005
2010
2015
2020
time
Figure SST-V2: mechanistic identifier reconstruction over long-term time series. Results are produced by concatenating subsequently predicted chunks.
20

---

180*
120
60"W
60
120
180"
60ºN
30"N
30"N
30'5
30'5
60'5
60'5
180*
120"W
120'E
180
-0.5
0.0
0.5
1.0
1.5
Figure 8: Example of global surface temperature in January, 1990.
Table 5: Training setup for sea surface temperature 6.3.
Non-applicable fields are marked with X.
Ada-GVAE
TI-MNN
Cont. Identifier
Mech. Identifier
Pre-process
DCT
DCT
DCT
DCT
Encoder
6-layer MLP
6-layer MLP
6-layer MLP
6-layer MLP
Decoder
6-layer MLP
6-layer MLP
X
3 proj. x 6-layer MLP
Time dim
208
208
208
208
State dim
1
1
Hidden dim
1024
1024
1024
1024
Latent dim
20
20
20
Optimizer
Adam
Adam
Adam
Adam
Adam: learning rate
le-5
1e-5
le-5
le-5
Adam: betal
0.9
0.9
0.9
0.9
Adam: beta2
0,999
0,999
0.999
0,999
Adam: epsilon
le-8
le-8
1e-8
le-8
Batch size
2160
2160
2160
2160
Temperature Alignment reg.
X
0.1
X
X
x
10
# Initial values
10
10
x
10
# Iterations
< 30,000
30,000
< 30,000
< 300000
Seeds
3
3
Latent regularizers in CRL. The framework proposed in $ can be generalized to many causal representation learning works, by specifying the latent regularizes according individual assumptions and settings. For example, in the multiview setting, the latent regularizer can be the L2 alignment between the learned representations on the shared partition eq. (3), as was assumed that the paired views are generated based on this overlapping set of latents [38, 65, 71]; in sparse causal representation learning the underlying generative process assumes observations are generated from sparse latent variables; therefore, the proposed algorithms actively enforce some sparsity constraint on learned representation [29, 31, 45, 70], provide more extensive summary of different causal representation learning approaches and their corresponding latent regularizer in Tab. 6. By replacing the alignment term (Cor. 3.2) with the specific latent constraints, one can plug in many causal representation learning algorithms construct identifiable neural emulator using framework.
21

---

Table 6: non-exhaustive summary of latent regularizers in recent CRL approaches.
Principle
Assumption
Latent regularizer
References
Locatello et al. [39], Von Kligelgen et [65]
9())sll [l(()ss
multiview
part. shared latents
Daunhawer al. [16], Yao al. [71]
llx(x) o(x) 61
Ahuja et al. [2]
Lachapelle al. [31], et et. [70]
sparse causal graph
sparsity
Spike and Slab prior
Moran et al. [45], Tonolini et al. [61]
temporal sparsity
KL (g(2f) | ==))16("")=5tta") Lachapelle and Lacoste-Julien [29]
Identifying time-varying parameters Time-varying parameters (()) could also be potentially iden- tified when they change sparsely time. For example, a time-varying parameter 0g remains constant between (fx. fR+1). Then, the states in between xt),, x(t+ ... x(t+ k) can be considered as mul- tiple views that share the same parameter Oz. Following this perspective, the time-invariant parameters considered in the scope of this paper remain consistent through the whole timespan (0, 4max, thus all discretized states x(41).... x(Ir) are views that share this parameter. This inductive bias is directly built into the architecture design by inputting the whole trajectory into the encoder instead of doing so step by step (where the time axis considered as batch dimension). From another angle, the time- varying parameters (t) could be interpreted explicitly defined differential equation, which gives rise to a partial observable setup. This direction has been studied in the context of sparse system identification without explicit identifiability analy- sis [40]. In a more general setting, time-varying parameters has been considered as latent trajectories and extensively studied in the field of temporal causal representation learning [33, 57, 72]. A more detailed related work section in this regard is provided in & 5 under CRL and dynamical systems.
hidden part of the state space vector x(t) without an
Model evaluation on real-world data. A great obstacle hindering causal representation learning scaling to real-worls data is that no ground truth latent variables are available. Since the methods aim to identify the latent variables, it is hard validate the identifiability theory without ground truth-generen ruth-generaainn factors. However, properly evaluating the CRL models on real-world data can be conducted by carefully designing causal downstream tasks, such climate zone classification and ATE estimation shown applied datasets, in $ 6.3. Overall, we believe by incorporating domain knowledge of the use CRL to answer important causal questions from individual fields, thus indirectly validating the identifiability
22

---

Table 7: Theorem 3.1 validation using ODEs with known functional form:
Experiments complex dynamical systems from ODEBench [15] and Cart-Pole (inspired by Yao et al. [72]), for exact parameter identification. RMSE is computed over 100 randomly sampled parameter groups (nearby chaotic configuration for chaotic systems) and averaged over the parameter dimension.
ID
System description
Equation
RMSE (m std)
RC-cireuit (charging capacitor)
8o
3e 2d:2
#2
2
Population growth (naive)
9p2g
3
Population growth with carrying capacity
9ozg
4e 5.t-2--- =B
4
RC-cireuit with non-linear resistor (charging
-0.5+
8c. 5db44 44
capacitor)
90
+1
5
Velocity of a falling object with air resistance
80 6148
Ge- 44
Autocatalysis with one fixed abundant chemi-
9px 01=0
9e 9e-5:ile5
cal
7
Compertz law for tumor growth
P6ag log (0
3e 22-24-4--2
8
Logistic equation with Allee effect
9oza (11--11--0 50
(1
8e 3e-3+-9e->
9
Language death model for two languages
90-(1-z0)
1e-11.3e-b
10
Refined language death model for two lan- guages
fomg 21 .(1 200 (1-000000000
1c- 2
11
Naive critical slowing down (statistical me- chanics)
X
12
Photons in laser (simple)
Po:000113 01=0
4e 4e-3:3e-3
13
Overdamped bead on a rotating hoop
8o(0) cos (aa) 1) (#o)
3e-44-7--5
14
Budworm outbreak model with predation
9pzza
(1- 6 o8
Se-3:199-4
15
Budworm outbreak with predation (dimen- sionless)
9pxra
(1- ) =
=8
4e 4e-5:15e-
16
Landau equation (typical time scale 1) Logistic equation with barvesting/lishing
9pang
0 9228
Ge 3d-3-1e-2
17
9pzg
02
Ge Ge-44-2e-4
18
Improved logistic equation with harvesting/- fishing
4x 24
19
Improved logistic equation with harvesting/- fishing (dimensionless)
01 +-gg (1 (1=
4k 5
20
Autocatalytic gene switching (dimensionless)
9 80-0, un 11
20-51le-5
21
Dimensionally reduced SIR infection model for dead people (dimensionless)
% 8yag =g
86 0e=622e-6
22
Hysteretic activation of protein expression (positive feedback. basal promoter expres- sion)
9
91 -0y20 02+=0
3e- 0e-24-2-22
23
Overdamped pendulum with constant driv- ing torppe/fincfies/lossppooo junction (di- mensionless)
90 (#o)
8e 86-668e-7
24
Harmonic oscillator without damping
5
-garo
25
Harmonic oscillator with damping
9e-4
26
Velte-Volteraa competition model (Strogatz version with sheeps and rabbits)
#o(4o (off--12#1=00)
7e-244ee2
=7(0p=mo-z1)
27
Lotka- Volterra simple (as on Wikipedia)
#0(4)-1141)
9--311e-3
(92-
28
Pendulum without friction
65-5:12--5
-o sin (zo)
29
Dipole fixed point
0g
2e 2e-3113-44
-
23

---

DD
System description
Equation
RMSE (m
std)
30
RNA molecules catalyzing each others repli-
#to f0000
7e-6:-e-7
cation
21 ( 11
31
SIR infection model only for healthy and sick
1
32
Damped double well oscillator
35-115e-5
-Poar1 -20 H-rg
-0px0 sin (xx))
33
Glider (dimensionless)
3e-1122-4
sou (211) xng
2g
34
Frictionless bead on a rotating hoop (dimen- sionless)
4e-5:22-5
(-Og+coo (rr)))
35
Rotational dynamics of an object in a shear flow
cos (sa) (#1)
3e 3e-313r-
(4o sin (xr) Hoow (x1)) sin (#o)
36
Pendulum with non-linear damping, no driv-
6e-4#le-
ing (dimensionless)
9pay cos (an) - -sin (Ho)
37
Van der Pol oscillator (standard form)
4e-456e-5
-0021 (2ā (10--1))20
o
38
Van der Pol oscillator (simplified form from Strogatz)
+xo+x1
2e-3ile-4
9g
Oazi +2]:+ -x0
39
Glycolytic oscillator, e.g. ADP and F6P in
8e-417e-5
yeast (dimensionless)
-6200 -f-11
40
Duffing equation (weakly non-linear oscilla- tion)
21
4g-441e-
09zn (1-28) -ag
0o(0+=20)
41
Cell cycle model by Tyson for interaction be- tween protein ede2 and cyclin (dimension- less)
1e=24-2e22
02-20
91g021 -20 +f+1
42
Reduced model for chlorine dioxide-iodine- malonic acid reaction (dimensionless)
3e 3::33
02
21 xp+1
43
Driven pendulum with linear damping / Josephson junction (dimensionless)
s1
Ge Ge-DiteeD
00-01x: -sin (xp)
44
Driven pendulum with quadratic damping (di- mensionless)
2
(1-xx)) =agaf
45
Isothermal autocatalytic reaction model by Gray and Scott 1985 (dimensionless)
=oa=
(agg 21) (1) sinn sin (#0)
46
Interacting bar magnets
16-6
9g sin (zo 1 -ii- (11)
47
Binocular rivalry model (no oscillations)
-20 -1
7e-4ble-4
-31 -o000-114
#021 Og 1108
48
Bacterial respiration model for nutrients and oxygen levels
4e-212e-2
02 91e8
49
Brussclator: hypothetical chemical oscillation model (dimensionless)
01=1=1 (2000++))1
26-2i5e-3
00
0a 0u+=fa1-zu
50
Chemical oscillator model by Schnackenherg 1979 (dimensionless)
Ir- 1e-644e-7
0 -1-1
51
Oscillator death model by Ermentrout and Kopell 1990
tiin (I11) cos (zo)
le 6496-8
0 tssn (41) cos (#o)
24

---

ID
System description
Equation
RMSE
std)
Ou(-zorta1)
52
Maxwell-BBoos equations (laser dynamics)
(egag ma1)
4e-2ide-2
+0g
R0-05
Pg
53
Model for apoptosis (cell death)
(1
86
07
1e-2i5e-3
-922200 N6+=1
0,51
09421
Ro(-20ta1)
54
Lorenz equations well-behaved periodic regime
01z0 0170-1022-22
1e-224e-3
-222+
0o (o(=2o+a1)
55
Lorenz equations in complex periodic regime
#1 mo 2000-00----00
8e-2i5e-2
-2222t0031
9o(=2o+ar)
56
Lorenz equations (chaotic)
0170-10#2-22
4e 2::9
-02
03(-Z1-F2)
57
Rössler attractor (stable fixed point)
0g(0)=1**p)
Se 2:
0x(0+=az(002tt0)
0s(=#1-F2)
58
Rössler attractor (periodic)
03(00=1*xp)
2e 2:
03(-z1-Ra)
59
Rössler attractor (chaotic)
05(00=1+*0)
4e-214ee-2
60
Aizawa attractor (chaotic)
86-412e-4
R020-2122
61
Chen-Lee attractor (chaotic)
0,
4e 2e=2+2e=2
02 2q
Foz2 16111 42
62
Binocular rivalry model with adaptation (os- cillations)
03(mp-21)
1e-3:12e-4
-xg
5040 10143 42
#3(m2-23)
-13002
63
SEIR infection model (proportions)
0p21+0:0002
41-55
02
mmml(aa siu(ce) con(a))
mpp
Cart-Pole (inverted pendulum)
asin(a) comt)
P+mpld in sin(n)
my
16-426e-
mp
mg + my
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide detailed proofs in App. B and successfully demonstrate our main claim in & with various experiments and datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations were discussed in $ 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
26

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions and theoretical statements are given in $ 3; proofs are provided in App. B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details about data generating pipeline, model architecture and training setup are provided in App. C for all baselines and experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
The wind simulation data used in 6.2 is generated from based open source library, which is given in App. C.I. Both code for generating this dataset and training are included in the supplementary material, following NeurIPS code and data submission guidelines. Curated code will be published upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please
the NeurlPS code and data submission guidelines (https://nips) httpp//...//gggddd///eeetaaaaaaa tails.
for more de-
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https://nig) ccc//ublic/guiiiieeeeeeeeepoeeee Saaees//ooouuumiiieeeccee peiiinnPolcc) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details are included in App. C.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results are computed from three independent runs, as given in Tabs. and 5.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute-related information is provided in App. C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
29

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpa//pp....pppppc//cccciiiiiee
Question:
Answer: [Yes]
Justification: The authors carefully reviewed the NeurlPS Code of Ethics and believe none the concerns from NeurIPS Code Ethics applies to this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on general representation learning: thus, there is immediate positive or negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The data model released by this paper do not have high risk of misuse.
30

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We carefully cited all datasets and works that were used in this paper. Also, the software we used to generate the simulation dataset open-sourced under an MIT license (see App. C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: This work focuses general representation learning and dynamical systems. Thus, we do not release new assets other than the source code to reproduce the experimental results. This code is attached to the submission and all corresponding training details are documented in App. C for reproducing.
Guidelines:
The answer NA means that the paper does not release new assets.
31

---

Researchers should communicate the details of the dataset/cooeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This work does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "6hb16SStHC", "reviewer": "Reviewer_bh3V", "summary": "This paper bridges causal representation learning (CRL) with dynamical system learning. \n\nIt introduces partially identifiable and practical models by merging methodologies from both CRL and dynamic systems. \n\nThe authors develop models capable of handling out-of-distribution classification tasks and treatment effect estimation. \n\nNotably, they validate their approach using a wind simulator and real-world climate data, effectively demonstrating the model's potential in addressing causal questions related to climate change.", "questions": "Please see Weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is one of the few causal representation learning works that claim contributions to theoretically sound methods with working and impactful real-world problem solutions.\n\n2. The authors provided a theoretically sound analysis of the partial identifiability of the model.\n\n3. The task chosen by the authors to validate their proposed method is very interesting and potentially impactful: climate model estimation. Solving such a problem and providing solutions with theoretical guarantees will be very impactful.", "weaknesses": "The proposed method claims partial identifiability of the invariant part of the dynamic models. There are existing works, such as [1], which study dynamic models in hidden space and demonstrate partial identifiability for the invariant part of the dynamic models. \n\nCan the authors provide some comments on why they chose to study the dynamic model in the observational space and how their proposed approach might be better than the existing work?\n\n\n\n[1] Li, Zijian, et al. \"When and How: Learning Identifiable Latent States for Nonstationary Time Series Forecasting.\" arXiv, 2024, arxiv.org/abs/2402.12767.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reply to weaknesses\n\nThank you for the positive feedback and for providing this interesting related work (listed as [`17`] under our references). Although our paper may **superficially resemble** [`17`], there are **important differences** between the two. Please allow us to make a quick clarification:\n\n1. The **hierarchy of the problem setting** differs: At a high level, [`17`] models dynamics in the latent space by treating **time-varying** latent variables as hidden states, while we model dynamics directly in the **observational** space. A discussion on this distinction is provided in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF) under \"Why not model dynamics in the latent space.\"  \n2. The **mixing process** differs: We consider the **entire trajectory** as our entangled **observation**, generated by $(x\\_{t\\_1}, …, x\\_{t\\_N}) \\= F(\\\\theta)$, where the **mixing** function $F$ is inherently **non-stationary** and depends on the timespan of interest. In contrast, the referred work [`17`] uses stationary diffeomorphism mixing and **implicitly** accounts for **non-stationary properties** within the latent space.  \n3. Our **definition of “stationary/invariant”** is different:  [`17`] considers a partition of latent variables $z\\_t^s$ as stationary when it is **invariant to the environment** $e\\_t$, meaning there is no causal link from the environment $e\\_t$ to $z\\_t^s$. However, $z\\_t^s$ still depends on time. In contrast, in our context, a stationary parameter is **time-invariant** and trajectory-specific, such as the pole length $l$, gravity $g$, cart mass $m\\_c$, and pole mass $m\\_p$ in the cart pole experiment (see `additional experiments` under [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF)).  \n4. Overall, we believe that [`17`] is more closely related to [`8,16`], and that both lines of work are **orthogonal yet equally important**. We will provide a thorough discussion of this connection in the updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQa9rJiTcm", "reviewer": "Reviewer_3NaZ", "summary": "This paper proposes a theory and methodology for representation learning from dynamical systems. In particular, it proposes a model in which latent causal variables deterministically generate an observed time-series trajectory through an ODE, with the task of identifying the latent variables from observed time series. Theoretically, the authors devise identifiability conditions by drawing the causal representation learning literature, treating the ODE as the (black box) mixing function. To identify variables in practice, it is proposed to use mechanistic neural networks (MNNs) to learn the ODE mixing function. Empirical results show that the proposed method can strike a good balance between identification of the latent variables and predictive performance.", "questions": "- The results show that one can identify the causal variables (up to blocks). Does this mean that the ODE function is also learned correctly, and thus we can trust e.g. the ATE in Experiment 6.2?\n- What is forecast error in Table 2? (MSE?)\n\nSuggestion on presentation: It can sometimes be confusing how $x, \\theta, $ and the parameters in $\\alpha$ relate. It might be useful to include a diagram (e.g. plate notation) to show which variables are time-dependent, trajectory-dependent etc.", "limitations": "The limitations of the method are mostly well-discussed in the text and conclusion. There are two other limitations I would encourage the authors to discuss. Firstly, as with most CRL methods, significant prior knowledge is required: in particular here, identifying twin trajectories where some unknown causal/latent parameters differ. Secondly, it is assumed that the ODE evolves over the observation space (e.g. in the sea surface temperature example, over the surface temperatures), whereas the true underlying ODE is more likely to involve other/latent temporal variables (e.g. amount of sunlight, polar ice cap coverage), on top of stationary latents.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic of the paper (learning interpretable causal variables from dynamical systems) is of great importance for e.g. the sciences and this paper makes a significant conceptual step in this direction. The specific proposed problem (recovery of stationary, trajectory specific parameters) and the idea of applying causal representation techniques (such as the multiview approach) to dynamical systems data is an interesting, and to my knowledge, novel problem and approach. \n\nThe proposed method for partially identifying parameters is a sensible combination of existing components (loss function from CRL literature, and MNN parameterization of ODE) that enables both accurate prediction of trajectories as well as identification of causal variables (by separating the causal variables $\\theta$ from the ODE parameters $\\alpha$).\n\nThe experiments are well-motivated and show the competitive/superior performance of the proposed method over baselines in terms of prediction and identification.\n\nThe paper is generally well written and has a clear structure, though some of the notation could be confusing at times (see suggestions below).", "weaknesses": "The novelty in terms of theory is fairly weak; the theoretical identifiability results directly follow from existing literature with minimal modifications, and does not exploit anything specific to ODEs (e.g. 1. are there certain ODE assumptions which enable identifiability, e.g. certain function types or sparsity in the ODE's equation; 2. how would the situation change with time-dependent latents)?\n\nIn terms of significance, the assumptions underlying the method are very strong and may limit the applicability of the method (e.g. determinism, no latent temporal variables).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and will address the concerns individually. \n\n&nbsp;\n\n### Reply to weaknesses\n\n1. Please refer to our general clarification on the **novelty and contribution** in [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF).  \n\n2. > are there certain ODE assumptions which enable identifiability, e.g. certain function types or sparsity in the ODE's equation?   \n   \n    **Identification of ODEs with certain properties** is discussed in [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF). Please refer to the paragraph  “Identifying ODEs with specific properties” for details.  \n\n3. > how would the situation change with time-dependent latents?\n\n   **Identification of time-varying latents** is discussed in `Appendix D`, and we will clarify this reference in the updated version, referencing it in the main text. This issue is closely related to the temporal CRL approaches that model dynamics in the latent space [`8, 16, 17`]. We offer a brief discussion of this in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF) under \"Why not model dynamics in the latent space\"  and will address the topic more thoroughly in the revised manuscript.  \n4. The validity and applicability of our **technical assumptions** are discussed in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF). \n\n&nbsp;\n### Reply to questions\n\n1. The ODE function is **learned implicitly** by the mechanistic neural network but with disentangled parameters, as evidenced by the **low forecasting error and OOD performance** (see `Table 2` in the main paper).  \n2. Thanks for this question. Yes, the forecasting error is the **mean squared error(MSE)** summed over the state dimension and averaged over batch and time dimensions. We will add this information in the updated manuscript.\n\n&nbsp;\n### Reply to suggestion on the representation\n\nWe thank the reviewer for this valuable advice. `Figure 6` in the Appendix illustrates such an overview of the architecture and relations between different variables (referred to in `lines 244-246`). We will clarify this link further in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AgUlWnRM1G", "reviewer": "Reviewer_aFgN", "summary": "The authors draw a connection between assumptions in the (neural) ODE and causal representation learning (more precisely, latent variable identifiability, but I will follow the authors and refer to it as CRL) literatures, by framing ODE inverse problems as latent variable problems. A particular focus is given to the case where the parametric form of the ODE is unknown, and only trajectories (i.e., time series data) are given. The learned generator and latent variables in CRL are analogous to a flexibly parametrized ODE solver and corresponding parameters, respectively. An example is given where pairs of trajectories sharing some parameter values are interpreted as same-content-different-style observations, which can be block-identified according to previous work in CRL. The authors describe a specific learning framework based on mechanistic neural networks which encode the trajectories into not only the parameters of a flexible family of ODEs, but also hyperparameters of the solver, defining a flexible generative process of observed trajectories.", "questions": "- (Repeated from above) Does the ATE estimation (l377) rely on identifiability, and would non-identifiable contemporaries fail in causal prediction? \n- The experiments also show improvements in classification, generalization, and forecasting---is there an explanation, even an intuitive one, for why CRL-identifiable models in particular might be preferable for these tasks?\n- Can the parameters learned by the pipeline always be interpreted as physical parameters of some underlying ground truth dynamics, or useful proxies thereof? Maybe if the MNN is well-specified?  \n\n### Other more minor points\n\n- Is it straight-forward to port over CRL results that rely on restricting the function class of the generator, e.g., orthogonal/sparse Jacobian, when it is supposed to represent an ODE solver? \n- The interpretation of dynamical systems as latent variable generative models may be the same approach taken in Bayesian inverse problems [1], for example see section 3.3 in the reference. If there is a connection here, maybe it should be discussed (disclaimer: I'm not at all an expert in Bayesian inverse problems). \n\n[1] Stuart, Andrew M. \"Inverse problems: a Bayesian perspective.\" Acta numerica 19 (2010).", "limitations": "As the paper does not introduce a new method but rather aims to draw connections, it is perhaps unnecessary to explicitly discuss the limitations. I think the authors do a fine job outlining potential future work in the final section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors state as motivation the need for successful real-world applications of CRL. This is an incredibly important problem to tackle, and cross-pollinating with the AI/ML for Science (specifically, inverse problems) community is a good place to start. There are many parallels between the areas of parameter identification in ODEs and latent variable identification in CRL: both recognize the significance of identifiability, but while it appears that application has outpaced theory in Neural ODEs, the opposite has occurred in CRL. The application of CRL theory to ODE inverse problems is hence original and potentially of great significance to both communities.", "weaknesses": "Although the paper draws mathematical connections between identifiability in inverse problems and CRL, I found the conceptual connection to be insufficient. The parameters in ODEs and latent variables in CRL, even if they can be made mathematically equivalent, are quite different in interpretation. The former represents physical parameters which need to be identified point-wise, whereas the focus in the latter (at least, within the scope of examples given in this paper) is typically in separating latent factors of variation, whereas identifiability of the factor itself can be up to arbitrary and unknown reparametrization (as long as it contains full and only information about that specific factor). For me, the lack of successful real world application of CRL is precisely because the community is unsure of what to do with these resulting factors, which have arbitrary units and are not physically interpretable. \n\nTo be fair, the usual ODE identifiability seems to be for the case where the dynamics are known, and not the \"ODE discovery\" setting that the paper focuses on in Sections 3.2+. Indeed, if all that is given are trajectories without an existing physical theory, it is clearly underdetermined to find physically meaningful parameters. Nonetheless, to build a truly useful bridge between the two fields, I find that the paper misses a crucial discussion on what this typical style of CRL identifiability, perhaps with unknown dynamics, can provide to the inverse problems community. For example, concerning the statement on l174:\n> identifying climate zone-related parameters from sea surface temperature data could improve understanding of climate change because the impact of climate change significantly differs in polar and tropical regions.\n\nThe paper would greatly benefit by giving more details on why CRL-style identifiability is important in studying this problem. For example, does the ATE estimation (l377) rely on identifiability, and would non-identifiable contemporaries fail in causal prediction? \n\nI really believe in the message of the paper and the mission to improve real-world applicability of CRL. However, unless the benefits of CRL-style identifiability for ODE inverse problems can be spelled out more explicitly, I do not believe the submission is ready for publication. Instead, I would strongly advocate for publication of a version where the following questions are at least partially answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reply to weaknesses\n\nWe appreciate the reviewer’s thoughtful comments. We concur that identifying parameters from an unknown system is highly **challenging** and can be **difficult to interpret**. We also acknowledge that the **CRL-identifiability** provided in `Corollary 3.2` is **limited**, as it isolates certain parameters only up to a bijection. However, given that point estimates for parameter identification in **unknown** systems are **fundamentally impossible** (`lines 198-200`), our neural emulator, which achieves **strong forecasting** performance and provides **some level** of parameter identification, surpasses models that focus solely on forecasting without considering identifiability, as shown by comparison to the **non-identified TI-MNN** model in `Table 2` in the main text. We argue that CRL-identifiability remains valuable for various downstream analyses, even when the underlying system is unknown:\n\n1. Identified latent parameters help **understand** **causal effects under intervention**. For example, in `Figure 6` (Appendix), the sea surface temperature drops when the inferred latitude-related variable $\\\\hat{\\\\theta}$ is permuted. This can be viewed as a **sensitivity analysis** within the dynamical system community, typically studied experimentally on a case-by-case basis.  \n2. Prior works [`19, 20, 21`] have demonstrated that **CRL-identified** latent variables **outperform** in related **downstream** classification tasks and exhibit **greater robustness** in **domain adaptation** **and out-of-distribution** scenarios. When these identifiable CRL algorithms are applied to **dynamical systems**, these advantageous properties are **retained**, as evidenced by `Figure 2` and `Table 2` in the main paper.  \n3. **Isolating covariates** information is crucial for treatment effect estimation, see the second point under `additional experiments` under [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF).\n\n&nbsp;\n\n### Reply to questions (same order as given by the reviewer)\n\n1. Thank you for this question. ATE estimation **critically depends on identifiability**. Our additional experiments demonstrate that **non-identified** parameters yield **meaningless** ATE estimates, whereas **identified** parameters provide an **increasing** trend. We will update this figure in the revised manuscript to reflect these findings. Please refer to the **second point** in `additional experiments` under [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF) for more details.  \n2. Intuitively, CRL-identified latent variables are valuable for downstream tasks because the learned representation accurately captures **all and only** the information about the ground truth parameters, **isolating confounding factors**, which is critical for treatment effect estimations (see `additional experiments`). For example, if a task is related to a parameter $\\\\theta$, a **non-identified** representation might include **arbitrary information unrelated** to $\\\\theta$. Using such representations for classification tasks could perform **no better than random guessing**. This is why we evaluate the learned representation on downstream classification tasks and robustness tests to validate its identifiability.  \n3. When the MNN (Mechanistic Neural Network [`22`]) is well-specified, the learned parameters can **correspond one-to-one** with physical parameters through a bijection. This bijection serves as a useful proxy in **sensitivity analysis** and provides **high-quality features** for downstream classification tasks, as previously discussed. If some physical parameters are known for certain trajectories, the **bijection** can be **learned explicitly**, allowing the mapping of the learned representation to the ground truth.\n\n&nbsp;\n\n### Reply to other minor points (same order as given by the reviewer)\n\n1. Thanks for raising this point. It is discussed in [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF) under “Identifying ODEs with specific properties.”  \n2. We agree that interpreting dynamical systems as latent variable models parallels approaches used in Bayesian inverse problems. As discussed in `lines 158-164`, **most Bayesian methods for parameter identification**, such as Gradient Matching [`4`], assume a **known** functional form but often **lack clear identifiability statements**. In contrast, our `Corollary 3.1` provides a theoretical framework that **explains** the successful empirical identification results observed with these methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JUMcxOXPjE", "reviewer": "Reviewer_yeqY", "summary": "This paper aims to connect causal representation learning with parameter identification in dynamical systems. By doing so, existing causal representation learning approaches can be used for estimating parameter in dynamical systems with identification guarantees. On the other side, it also demonstrates the applicability of causal representation learning on real-world data. Experimental evaluation on simulated and real-world climate data show the effectiveness of applying causal representation learning approaches in solving parameter identification tasks in dynamical systems.", "questions": "1: Are the assumptions (e.g., assumption 2.1 and 2.2) wild? Given which properties the dynamic systems can satisfy these assumptions? \n\n2: How is your work different from [a], which can deal with videos and noisy data under identification guarantee?\n\n3: Why you use mechanistic neural networks? \n\n4: How does the proposed approach handle heterogeneous data with multiple sets of parameters?", "limitations": "The authors have discussed the limitations of the work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper focuses on an intriguing and emerging topic on parameter identification in dynamic systems. Particularly, the authors attempt to leverage existing causal representation approaches to address the parameter identification problem. Re-formulating parameter identification problem as a causal representation learning problem is an appealing insight.", "weaknesses": "1: Contribution is not significant. Though the paper aims to set up the connection between CRL and parameter estimation, most of the discussions are based on CRL. This work is more like an application of CRL to parameter estimation in dynamical systems. The identifiability theories are mostly adopted from CRL. It is hard to understand their practical effectiveness in solving parameter identification task in dynamic systems. Providing specific dynamic systems as examples can be helpful in understanding the usage of these theories.\n\n2: Missing important related works. Parameter identification has been addressed with identifiability guarantee, e.g., [a], while these works are not well discussed. \n\n[a] Yao, W., Chen, G., & Zhang, K. (2022). Temporally disentangled representation learning. Advances in Neural Information Processing Systems, 35, 26492-26503.\n\n3: Experiments are insufficient. Only simulated wind data and real-data on sea surface temperature are considered. To have a comprehensive understanding of the effectiveness in dynamic systems, different types of dynamic systems should be considered. Ablation studies should be performed on scenarios where true functional form is known for verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reply to weaknesses (same order as given by the reviewer)\n\n1. The [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF) clarifies our paper’s **novelty and contribution**. Additionally, we successfully demonstrated parameter identification in various existing systems. Please refer to `additional experiments` in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF).   \n2. We thank the reviewer for providing this line of related work ([a] is listed as [`8`] under our references). Our paper looks **superficially similar** to those, but they are **conceptually different**. Please allow us a quick clarification in this regard:  \n   1. Reference [`8`] models dynamics in the **latent space**, whereas our approach models dynamics directly in the **observational** space. The reasons for this choice are detailed in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF).  \n   2. Consequently, our framework addresses **a different hierarchy of problems**. For instance, in the cart-pole example from Figure 3 in [`8`], while [`8`] treats the time-varying ODE states (**cart position, pole angle**) as latent variables, our focus is on estimating higher-level, time-invariant parameters such as **gravity, cart mass, pole mass, and pole length** for the known system (see `additional experiments`). A similar distinction applies to the motion capture dataset in [`8`].  \n   3. Hence, we believe that both problems are **orthogonal yet equally important, and we encourage cross-pollination in future work**. We will thoroughly discuss these works in the updated manuscript and explore the potential applications of these approaches to scientific discoveries.  \n3. **Additional studies** are provided on systems with **known functional forms**, including the **cart-pole** system inspired by [`8`]. Please refer to `additional experiments` for details. We thank the reviewer for this suggestion and kindly ask them to reconsider their scores in light of the new experiments. While we cannot directly compare against [`8`] because we take as input the whole trajectory instead of modeling the dynamics step by step, we will carefully explain the differences and opportunities for synergy between the approaches. \n\n&nbsp;\n### Reply to questions (same order as given by the reviewer)\n\n1. The **validity of the assumptions** is discussed in the [global rebuttal](https://openreview.net/forum?id=MWHRxKz4mq&noteId=n59QJ5gxJF).  \n2. The **difference** between our paper and the **provided reference** is discussed in the second point of “Reply to weaknesses.”  \n3. The reason **why we use MNN** is discussed in Appendix D (`lines 727-739`). We will refer to this link more clearly in the updated manuscript, and we apologize that this was not clearly referenced in the main text of the submitted draft.   \n4. Thank you for this intriguing question. It is indeed **possible** to identify latent parameters from **heterogeneous** time series data using multiview CRL algorithms [`6`], which assume multimodal mixing functions for different views. This paper primarily focuses on a single climate dataset to **establish a connection** between causal representation learning and dynamical systems; thus, the consideration of heterogeneous data is **outside** **the** **scope** of this study.  However, we acknowledge that extending this approach to **heterogeneous** data is a **promising** area for future research. Broadly, we believe that many algorithms developed in causal representation learning can be adapted for dynamical systems, and we hope our paper lays the **foundations** for and **encourages** this future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are extremely grateful to the reviewers and AC for their time and valuable feedback. We very much appreciate that they found the problem we are tackling is *“very interesting and potentially impactful*”(`bh3V`), “*of great importance for e.g. science*”(`3NaZ`), “*incredibly important*”(`aFgN`), and “*intriguing and emerging*”(`yeqY`). We are happy to see that our idea of applying CRL techniques to dynamical systems is *“interesting and novel”* (`3NaZ`), “*original and potentially of great significance to both communities*”(`aFgN`). 

**Novelty and contribution (`3NaZ`, `yeqY`)**
* As discussed in `lines 73-78`, this paper’s main contribution is providing **clear parameter-identifiability statements** for dynamical systems, whereas numerous **previous works** on ODE discovery [`1, 2, 3, 4`] refrained from doing so by explicitly stating that it is **unknown** which settings yield identifiability (`lines 165-166`).  
* To the best of our knowledge, our paper is the **first** to target **real-world scientific applications** and successfully demonstrate identification results on **raw measurements**, unlike **previous CRL** work which relies on synthetic or **heavily pre-processed** data (e.g. manually rendered images[`5, 6`] or extracted avatar skeleton [`9`]).

**Validity of the technical assumptions (`3NaZ`, `yeqY`)**
* Assumptions 2.1 and 2.2 are **standard and necessary** for parameter identification in dynamical system [`10, 11`]. Many ODEs satisfy these assumptions, including Lotka-Volterra ODE, Van der Pol oscillator, and chaotic systems such as the Lorenz attractor and the Rössler attractor. 
* Further, we justify in `Table 1` that **CRL assumptions** (`3.1-3.3`) **align** with standard assumptions 2.1 and 2.2, establishing the theoretical ground for cross-pollination between these two fields.  
* Nevertheless, we acknowledge the limitation of the determinism assumption, as discussed in `Section 6`.

**Identifiability with ODE with specific properties (`3NaZ`, `aFgN`, `yeqY`)**
* `Corollary 3.1` provides full identifiability for stationary parameters in ODEs with known functional forms, including **any parametric form** (linear, polynomial, even nonlinear). This is **supported** by various empirical studies from **prior literature** on equation discovery [`1, 2, 3, 4`] and attached **additional experimental results**.  
* For ODEs with partially known properties, such as a **sparse linear combination** $f_{\theta}(x, t) = \sum\_{i = 1}^{m} \theta_i \phi_i(x)$  **of various basis functions** $\phi_i$ within a comprehensive dictionary (e.g., SINDy-like scenarios [`1, 2, 3`]), `Corollary 3.1` ensures parameter identifiability under a sparsity constraint. Notably, the ODE relates to the parameters $\theta$ linearly, as detailed in the main paper (`lines 154-157`) and the appendix (`lines 653-663`).  
* **Many existing CRL works** assume certain properties on the generating process, such as **sparsity** [`14`] or **specific functional class** [`15`], which can be **directly imported** into our framework by replacing the multiview approach.  
   

**Why not model dynamics in the latent space (`bh3V,3NaZ,yeqY`)**
* We model dynamics directly in the observational space because the underlying parameters $\theta$ **directly determine** the whole trajectory: $(x_{t_1}, \dots, x_{t_N}) = F(\theta)$.   
* **Almost all CRL works** assume the latent variables **directly influence** the observation, except for a few considering hierarchical latent models [`12, 13`], which we will thoroughly discuss in the updated manuscript.  
* As discussed in `Section 6`, we agree assuming direct observation of the states is limiting, and relaxing this assumption is an interesting future direction. However, since this paper is the **first to formally connect** CRL-identifiability with inverse problems in dynamical systems, we believe that exploring this perspective is beyond the scope of the current study.


**Additional experiments (see attached pdf)**
* We provide additional experiment results on specific dynamical systems with **known functional forms**, including eight highly complex systems from **ODEBench**[`18`] and the **Cart-Pole** system inspired by [`8`].   
  * For each system, we sample 100 tuples of parameters $\theta$ within a valid range to preserve properties like chaos. Each problem is solved exactly as described in `Corollary 3.1`.   
  * The root-mean-square deviation (RMSE) is averaged across the parameter dimensions and sample size (100), reported via mean and std.
  * We observe **highly accurate point estimates** for all stationary system parameters $\theta$, validating `Corollary 3.1` **across various experimental settings**. Due to space constraints, we cannot enclose results for all 63 systems from ODEBbench[`18`]. Complete results and implementation details will be provided in the revised manuscript. 
* Comparing **ATE** estimates from non-identified and identified representations for SST-V2:   
  * The attached `Figure 1` illustrates the estimated ATE from the **non-identified** representation **lacks a discernible pattern**, while the **identified** one exhibits a noisy yet **clear increasing trend**, indicating the global warming effect.   
  * This is because the non-identified representation failed to isolate the covariates $\theta$, leading to biased treatment effect estimates. To estimate treatment effects, the **covariates** (i.e., the latitude-related parameters we identify) **mustn’t be influenced by the treatment** (i.e., the climate zones). Otherwise, they become confounders, leading to incorrect estimates [`22`].   
  * We apologize for not explaining this clearly in the current draft. In the final version, we will add the additional graphical model and the non-identified baseline (attached `Figure 1`) together with a thorough explanation about treatment effect estimation with covariates and why CRL-identifiability (up to bijection) is necessary to avoid confounding.

Meta-review: This paper proposes a connection between causal representation learning and dynamic system, ODE, to establish the identifiability of the dynamics system. Reviewers recognise the contribution of the paper, although the technical novelty is limited. During the rebuttal period, the author's responses addresses many concerns from the reviewers, so it is recommended to include these in the revised version.
Decision: Accept (poster)