Self-supervised contrastive learning performs non-linear system identification
Authors: Rodrigo González Laiz
Tobias Schmidt
Steffen Schneider
Venue: ICLR 2025 Poster (2025)
Keywords: system identification
dynamics learning
identifiability
self-supervised learning
Primary area: learning on time series and dynamical systems

Abstract: Self-supervised learning (SSL) approaches have brought tremendous success across many tasks and domains. It has been argued that these successes can be attributed to a link between SSL and identifiable representation learning: Temporal structure and auxiliary variables ensure that latent representations are related to the true underlying generative factors of the data. Here, we deepen this connection and show that SSL can perform system identification in latent space. We propose dynamics contrastive learning, a framework to uncover linear, switching linear and non-linear dynamics under a non-linear observation model, give theoretical guarantees and validate them empirically.

Published as a conference paper at ICLR 2025
SELF-SUPERVISED CONTRASTIVE LEARNING PERFORMS NON-LINEAR SYSTEM IDENTIFICATION
Rodrigo González Laiz; Tobias Schmidt & Steffen Schneider!
Institute of Computational Biology, Computational Health Center, Helmholtz Munich and Munich Center for Machine Learning (MCML)
ABSTRACT
Self-supervised learning (SSL) approaches have brought tremendous success across many tasks and domains. It has been argued that these successes can be attributed to a link between SSL and identifiable representation learning: Temporal structure and auxiliary variables ensure that latent representations are related the true un- derlying generative factors of the data. Here, we deepen this connection and show that SSL can perform system identification in latent space. We propose dynamics contrastive learning, a framework to uncover linear, switching linear and non- non-linear dynamics under non-linear observation model, give theoretical guarantees and val- idate them empirically. Code: gi github. com/ wwt.ccmmmmmn/eeeeeee dynami cal inffrrenee/dd.....
INTRODUCTION
The identification and modeling of dynamics from observational data is Long-standing problem in machine learning, engineering and science. A discrete- time dynamical system with latent variables æ, observable variables y, control signal u, its control matrix B, and noise E, v can take the form
T++1 f(x:)= Bu: Et ( Mi
and we aim to infer the functions f and g from a time-series of observations and, when available, control signals. Numerous algorithms have been developed to tackle special cases of this problem formulation, ranging from classical system identification methods (McGee Schmidt, 1985; Chen Billings, 1989) to recent generative models (Duncker et al., 2019; Linderman et al., 2017: Hälvä et al., 2021). Yet, remains an open challenge to improve the generality, interpretability and efficiency of these inference techniques, especially when f and g non-linear functions.
Contrastive learning (CL) and next-token prediction tasks have become important backbones of modern machine learning systems for learning from sequential data, proving highly effective for building meaningful latent representations (Baevski et al., 2022; Bommasani et al., 2021; Brown, 2020; Oord al., 2018: LeCun, 2022; Sermanet al., 2018; Radford al., 2019). An emerging view is a connection between these algorithms and learning of world models (Ha & Schmidhuber, 2018; Assran al., 2023; Garrido al., 2024). However, the theoretical understanding of non-linear system identification by these aoguence-learning algorithms remains limited.
In this work, we revisit and extend contrastive learning in the context of system identification. We uncover several surprising facts about its out-of-the-box effectiveness in identifying dynamics and unveil common design choices in SSL systems used in practice. Our theoretical study extends identifiability results (Hyvarinen & Morioka, 2016; 2017: Hyvarinen et al., 2019; Zimmermann et al., 2021; Roeder et al., 2021) for CL towards dynamical systems. While our theory makes several predictions about capabilities of standard CL., it also highlights shortcomings. To overcome these and enable interpretable dynamics inference across a range of data generating processes, we propose a general framework for linear and non non-linear system identification with CL (Figure 1).
Background. An influential motivation of our work Contrastive Predictive Coding (CPC; Oord al., 2018). CPC can be recovered as a special case of our framework when using RNN dynamics
"Equal contribution.
'Coresponddecc: neffen..chneideen hwth://cecceieehhhhht.mmmmmiei.e hemmhoht---omhcch.eee

---

Published as a conference paper at ICLR 2025
model. Related works have emerged across different modalities: wav2vec (Schneider et al., 2019), TCN (Sermanet et al.. 2018) and CPCv2 (Henaff, 2020). In the field of system identification, notable approaches include the Extended Kalman Filter (EKF) (McGee & Schmidt, 1985) and NARMAX (Chen & Billings, 1989). Additionally, several works have also explored generative models for general dynamics (Duncker et al., 2019) and switching dynamics, e.g. iSLDS (Linderman al., 2017). In the Nonlinear ICA literature, identifiable algorithms for time-series data, such as Time Contrastive Learn- ing (TCL; Hyvarinen & Morioka, 2016) for non-stationary processes and Permutation Contrastive Learning (PCL; Hyvarinen & Morioka, 2017) for stationary data have been proposed, with recent advances like SNICA (Halvä al.. 2021) for more generally structured data- gata--gnnrating processes. In contrast to previous work, we focus on bridging time- series representation learning through contrastive learning with the identification of dynamical systems, both theo- retically and empirically. Morcover, by not relying on an explicit lata-cenerating model, our framework offers greater flexibility. We extend and discuss the connections to related work in more detail in Appendix C.
Zt
(dynamics)
f (encoder)
h
h
Contributions. We extend the existing theory on con- trastive learning for time series learning and make adap- tations to common inference frameworks. We introduce our CL variant (Fig. 1) in section 2, and give an identifi- ability result for both the latent space and the dynamics model in section 3. These theoretical results are later em- pirically validated. We then propose a practical way to parameterize switching linear dynamics in section and demonstrate that this formulation corroborates our theory for both switching linear system dynamics and non-linear dynamics in sections 5-6.
yt++1
y
Figure DCL framework: The encoder / shared across the reference yt. positive gr-1. and negative samples y, A dynamics model f forward predicts the reference. A (possi- bly latent) variable can parameterize the dynamics (cf. $ 4) external control (ef. $ The model fits the InfoNCE loss (L).
CONTRASTIVE LEARNING FOR TIME-SERIES
In contrastive learning, we aim to model similarities between pairs of data points (Figure 1). Our full model is specified by the log-likelihood
log (ylyt, ",,,, .N) (v log
Σ exp(1 y ENU(W!)
where y is often called the reference or anchor sample, y* is a positive sample, y E N are negative examples, and V is the set of negative samples. The model v itself is parameterized composition of an encoder, a dynamics model, and similarity function and will be defined further below. We fit the model by minimizing the negative log- likelihood on the time series,
min [iv]= min E E,,t,.... log pa (3441181, (ye... J. m==
where positive examples are just adjacent points in the time-series, and M negative examples sampled uniformly across the dataset. U U11, T') denotes a uniform distribution across the discrete time steps.
To attain favourable properties for identifying the latent dynamics, we carefully design the hypothesis class for vb The motivation for this particular design will become clear later. To define the full model, composition of several functions is necessary. Recall from Eq. 1 that the dynamics model is given as f and the mixing function g. Correspondingly, our model composed of the encoder h R + Rd (de-mixing), the dynamics model f. : Rd + Rd, the similarity function Φ : Rd x IRd + IR and correction term a Rdd + R. We define their composition as?
<(y.v') = ((f((())) h(y')) - a(y').
(4)
and call the resulting algorithm dynamics contrastive learning (DCL). Intuitively, obtain two observed samples (y. y') which are first mapped to the latent space, (h(y), h(y')). Then, the
Note that we equivalently write ((h())), h hl(x²)) using two asymmetric encoder functions, see addi- tional results in Appendix D.
2

---

Published as a conference paper at ICLR 2025
b
La b
y
g
hog
Figure 2: Graphical intuition hehind Theorem 1. (a), the ground truth latent space is mapped to observables through the injective mixing function Our model maps back into the latent space. The composition of mixing and de- de-mixing by the model affine transform. (b), dynamics in the ground-truth space mapped the latent space. By observing variations introduced by the system noise e, our model able infer ground-truth dynamics up to an affine transform.
dynamics model is applied to h(y), and the resulting points are compared through the similarity function Φb The similarity function & will be informed by the form of (possibly induced) system noise Eg. In the simplest form, the noise can be chosen as isotropic Gaussian noise, which results in negative squared Euclidean norm for Φ.
Note, the additional term (gy") is correction applied to account for non-uniform marginal distri- butions. It can be parameterized around the datapoints. In very special cases, the KDE makes difference in empirical performance (App. B, Fig. 9) and required for our theory. Yet, we found that on the time-series datasets considered, it was possible to drop this term without loss in performance (i.e., a(y') = 0).
kernel density estimate (KDE) with log ((((y') log q(x')
3 STRUCTURAL IDENTIFIABILITY OF NON-LINEAR LATENT DYNAMICS
We now study the aforementioned model theoretically. The key components of our theory along with notion of linear identifiability (Roeder et al., 2021; Khemakhem al., 2020) visualized in Figure 2. We are interested in two properties. First, linear identifiability of the latent space: The composition of mixing function g and model encoder h should recover the ground-truth latents up to linear transform. Second, identifiability of the (non-linear) dynamics model: We would like to relate the estimated dynamics f to the underlying ground- dynamics f. This property is also called structural identifiability (Bellman Aström, 1970). Our model operates on a subelass of Eq. with the following properties:
Data-generating process. We consider a discrete-time dynamical system defined as
T141 f f(x:) + Et, Y/ =xxx)
(5)
where E IRd are latent variables, f: _ + d is a bijective dynamics model, e, RRd the system noise, and g : Rdd + RD is non-linear injective mapping from latents to observables ye E IR.. d S D. We sample total number of T time steps.
We proceed by stating our main result:
Theorem (Contrastive estimation of linerr dynamics). Assume that
(Al) A time-series dataset {yo]T= is generated according to the ground-trut truth dynamical
system in Eq. 5 with bijective dynamics model and injective mixing function g.
(A2) The system noise follows an iid normal distribution, p(e.) = =N(e, Eg).
(A3) The model vp is composed of an encoder h, dynamics model f. correction term and the similarity metric o(u, v) - -1|u
v1l2 and attains the global minimizer of Eq. 3.
Then, in the limit T + 0o for any point x in the support of the data marginal distribution:
(a) The composition of mixing and de-mixing h(g(x)) Lx + b a bijective affine transform. and L = QE. 1/2 with unknown orthogonal transform Q E Iddxd and offset b E Rd. (b) The estimated dynamics f are bijective and identify the true dynamics f up to the relation
f(x) (L (xx b.
Proof. See Appendix A for the full proof, and see Fig. for graphical intuition of both results. n
3

---

Published as a conference paper at ICLR 2025
With this main result in place, can make statements for several systems of interest; specifically linear dynamics in latent space:
Corollary 1. Contrastive learning without dynamics model, f(a:) - cannot identify latent dynamics.
In this case, even for a linear ground truth dynamics model, f(x) = A., we would require that after model fitting, f(x)=
LAL- + b, which is impossible (Theorem lb; also see App. Eq. 22). We can fix this case by either decoupling the two encoders (Appendix D), or taking a more structured approach and parameterizing dynamics model with a dynamics matrix:
Corollary For a ground-truth linear dynamical system f Ax and dynamics model f(x) =
Ax, we identify the latents up to h(g(z)) = Lx + b and dynamics with Å LAL
This means that simultaneously fitting the system dynamics and encoding model allows us to recover the system matrix up to an indeterminacy.
Note on Assumptions. The required assumptions are rather practical: (AI) allows for a very broad class of dynamical systems as long as bijectivity of the dynamics model holds, which is the case of many systems used in the natural sciences. We consider dynamical systems with control signal u, in Appendix I. While (A2) is very common one in dynamical systems modeling, it can be seen more strict: We either need knowledge about the form of system noise, or inject such noise. We should note that analogous to the discussion of Zimmermann et al. (2021), it is most certainly possible to extend our results towards other classes of noise distributions by matching the oog-denstyy of æ with 6. Given the common use of Normally distributed noise, however, limited the scope of the current theory to the Normal distribution, but show vMF noise in Appendix D. (A3) mainly concerns the model setup. An apparent limitation of Def. 5 is the injectivity assumption imposed on the mixing function g. In practice, partially observable setting often applies, where g(x) C' maps latents into lower dimensional observations has lower rank than there are latent dimensions. For these systems, we can ensure injectivity through time-lag embedding. See Appendix H for empirical validation.
4 V-SLDS: TOWARDS NON-LINEAR DYNAMICS ESTIMATION
Piecewise linear approximation of dynamics. Our theoretical results suggest that contrastive learning allows the fitting of non- linear bijective dynamics. This is a compelling result, but in practice it requires the use of a powerful, yet easy to parameterize dynamics model. One option to use an RNN (Elman, 1990: Oord et al., 2018) or a Transformer (Vaswani, 2017) model to perform this link across timescales. An alternative option is to linearize the system, which we propose in the following.
18,
Backeropp via Currail Setteres
Warn
arg max [|Fee WoalF
We propose a new forward model for differentiable switching linear dynamics (V-SLDS) in latent space. The estimation is outlined in Figure 3. This model allows fast estimation of switching dynamics and can be easily integrated into the DCL algorithm. The dynamics model has a trainable bank W = [W1.... WK] of possible dynam- ics matrices. K is hyperparameter. The dynamics depend a latent variable k; and are defined as
MS W
Figure 3: The core components of the V-SLDS model parameter- free, differentiable parameteriza- tion of the switching pracess.
f(:: W. kx) = Wa, he = argming WAT, 21+1ll?. (6)
Intuitively, the predictive performance of every available linear dynamical system is used to select the right dynamics with index hn from the bank W. During training, we approximate the argmin using the
trick (Jang et al., 2016) without hard sampling:
f( W, 21) = 21,6 Wa)aa 22, = exp(A</T)
K
A& = |4 x1+1 1|
Exex(A)//7)
Note that the dynamics model f(x;; w, depends on an additional latent variable 2, [2t.1g---,, [ which contains probabilities to parametrize the dynamics. During inference,
4

---

Published as a conference paper at ICLR 2025
obtain the index X2- arg max; 2t,k- The variables ge samples from the Gumbel distri- bution (Jang et al., 2016) and we use a temperature T to control the smoothness of the resulting probabilities. During pilot experiments, we found that the reciprocal parameterization of the logits outperforms other choices for computing an argmin, like flipping the sign.
From linear switching to non-linear dynamics. Non-linear system dynamics of the general form in Eq. 5 can be approximated using our switching model. We can approximate a continuous non-linear dynamical system with latent dynamics - ((x) around reference points { using a first-order Taylor expansion, f()) x f(x)) = f(*)) + y (#x) #&), where we denote the Jacobian matrix of f with Jy. We evaluate the equation at each point t using the best reference point Tk. We obtain system matrices As- = J)(ā)) and bias term by = f(xx) y (ãx) which can be modeled with the V-SLDS model f fax;kk)
x'L- = (Ax, 2x + + f(1;; kig) E
(8)
While a theoretical guarantee for this general case beyond the scope of this work, we give an empirical evaluation on Lorenz attractor dynamics below. Note, as the number of "basis points" of V-SLDS approaches the number of time steps, we could trivially approach perfect estimation capability of the latents as store the exact value of f at every point. However, this comes the expense of having less points to estimate each individual dynamics matrix. Empirically, we used 100-200 matrices for datasets of 1M samples.
EXPERIMENTS
To verify our theory, we implement a benchmark dataset for studying the effects of various model choices. We generate time-series with IM samples, either single sequence or across multiple trials. Our experiments rigorously evaluate different variants contrastive learning algorithms.
Data generation. Data is generated by simulating latent variables that evolve according dynamical system (Eq. 5). These latent variables are then passed through a nonlinear mixing function g to produce the observable data y. The mixing function g consists of nonlinear injective component which is parameterized by a randomly initialized 4-layer MLP (Hyvarinen & Morioka, 2016), and a linear map to a 50-dimensional space. The final mixing function defined as their composition. We ensure the injectivity of the resulting function by monitoring the condition number of each matrix layer, following previous work (Hyvarinen Morioka, 2016; Zimmermann et al., 2021).
LDS. We simulate 1M datapoints in 3D space following fxx)) = Ax, with system noise standard deviation o, = 0.01 and choose A be an orthogonal matrix ensure stable dynamics with all eigenvalues equal to 1. We do so by taking the product of multiple rotation matrices, one for each possible plane to rotate around with rotation angles being randomly chosen to be -5 or 5
SLDS. We simulate switching linear dynamical systems with f(z;; kt) = As, x; and system noise standard deviation o, 0.0001. We choose A& to be an orthogonal matrix ensuring that all eigenvalues are 1, which guarantees system stability. Specifically, we set A to be rotation matrix with varying rotation angles (5%, 10", 20%). The latent dimensionality is 6. The number of samples is IM. We use 1000 trials, and each trial consists of 1000 samples. We use k 0, 1, K distinct modes following a mode sequence i, The mode sequence in follows a Markov chain with a symmetric transition matrix and uniform prior:
Cat(iT), where π. Ω for all At each time step, 4 ~ CTT( ). where IT is a transition matrix with uniform off-diagonal probabilities set to 10-4, Example data is visualized in Figure 4 and Appendix E.
Non-linear dynamics. We simulate 1M points of a Lorenz system, with equations
f(x:) v. dt æ1,t), (1 x3.z) x2.)), (01, Bo3,))
(9)
with varying dt, parameters a = 10, 3 - = 28 and system noise standard deviation a, = 0.001. The observable data, y. We then apply our non-linear mixing function as for other datasets.
Model estimation. For the feature encoder h, baseline and our model use an MLP with three layers followed by GELU activations (Hendryeks & Gimpel, 2016). Model capacity scales with the embedding dimensionality d. The last hidden layer has 10d units and all previous layers have 30d units. For the SLDS and LDS datasets, train batches with 2048 samples each (reference
5

---

Published as a conference paper at ICLR 2025
Table 1: Overview about identifiability of latent dynamics for different modeling choices: We show different data generating processes characterized by the form the ground truth dynamics f. the distribution p(e) and different model choices for the estimated dynamies We compare identity dynamics, linear dynamics (LDS), switching linear dynamics (SLDS), and Lorenz attractor dynamies (Lorenz), and optionally initialize the dynamics model with the ground-truv truth dynamics (GT), For every combination we indicate whether we can provide theoretical identifiability guarantees ("theory") and compare this to empirical identifiability measures R2, LDD,, dynR²). Mean + std. across 3 datasets (5 for Lorenz) and 3 experiment repeats.
Data
Model f
Results
S
p(e)
identifiable
%R²
LDS [x10
identity
Normal Normal
identity
99.56 1 0.21
0.00 1 0.00
identity
LDS
99.31 + 0.43
0.04 1 0.01
LDS (low At)
Normal (large σ))
identity
89.22 4.47
8.53 0.05
LDS
Normal
identity
73.56 24.45
21.24 0.31
LDS
Normal
LDS
99.03 + 0.41
0.77 + 1.07
LDS
Normal
GT
99.46 + 0.39
0,44 + 0.43
%H² 1
%dynk? 1
SLDS
Normal
identity
76.80 + 7.40
85.47 + 8.07
SLDS
Normal
V-SLDS
99.52 + 0.05
99.93 0.01
SLDS
Normal
GT
99.20 + 0.10
99.97 0.00
Lorenz (small Af)
Normal (large (7)
identity
99.74 + 0.36
99.94 0.07
Lorenz (small At)
Normal (large 07)
LDS
98.31 + 2.55
97.21 5.90
Lorenz (small At)
Normal (large g)
V-SLDS
94.14 + 4.34
94.20 + 6.57
Lorenz
Normal
identity
40.99 + 8.58
27.02 8.72
Lorenz
Normal Normal
LDS
81.20 1 16.93
80.30 1 14.13
Lorenz
V-LLD
(V)
94.08 + 2.75
93.91 5.32
and positive). We use 2 = 65536 negative samples for SLDS and 20k negative samples for LDS data. For the Lorenz data, optimizer (Kingma, 2014) with learning rates 3 x 10-4 for LDS data, 10-3 for SLDS data, and 10 for Lorenz system data. For the SLDS data, we use different learning rate of 10-2 for the parameters of the dynamics model. We train for 50k steps on SLDS data and for 30k steps for LDS and Lorenz system data. Our baseline model is standard self-supervised contrastive learning with the InfoNCE loss, which is similar to the CEBRA-time model (with symmetric encoders, i.c., without a dynamics model; cf. Schneider al., 2023). For DCL, we add an LDS or V-SLDS dynamics model for fitting. For our baseline, we post-hoc fit the corresponding model on the recovered latents minimizing the predictive mean squared error via gradient descent.
batch size of 1024 and 20k negative samples. We use the Adam
Evaluation metries. Our metrics are informed by the result in Theorem I and measure empirical identifiability up to affine transformation of the latent space and its underlying linear or non-linear dynamics. All metrics are estimated on the dataset the model is fit on. See Appendix F for additional discussion on estimating metrics on independently sampled dynamics.
To account for the affine indeterminacy, we estimate L, b for a Lx + b which allows to map ground truth latents æ to recovered latents & (cf. Theorem la). In cases where the inverse transform L (i b) is required, we can either compute L- directly, or for the purpose of numerical stability estimate it from data, which we denote as L'. The values of L.b and L' ,b' are computed via linear regression:
min L,
c (La,, + b)11
and
min L'.
x - (L'è, + b')l2
(10)
To evaluate the identifiability of the representation, we measure the R² between the true latents æ; and the optimally aligned recovered latents L' i; + b' across time steps t = 1... in the time-serics.
Not explicitly shown, hut the argument in Corollary 2 applies to each piecewise linear section of the SLDS. V-SLDS only approximation of the functional form of the underlying system.

---

Published as a conference paper at ICLR 2025
10
Cluster Accuracy
0.
0.9
0.8
07-
0.
dynamics
05
+ 0=0.001
denilty (6)
0=0.0001
3T SLOS
04
0.6
SLDS
03
5
10
20
Rotation Angle (Degrees)
Rotation Angle (Degrees)
Figure 4: Switching linear dynamics: (a) example ground-truth dynamics in latent space for four matrices Ag- (h) R metric for different noise levels as we increase the angles used for data generation. We compare baseline (no dynamics) V-SLDS and model fitted with ground-truth dynamics. (c) cluster accuracies for models shown (b).
We also propose two metrics as direct measures of identifiability for the recovered dynamics f. For linear dynamics models, introduce the LDS error. It denotes the norm of the difference between the true dynamics matrix A and the estimated dynamics matrix Á by accounting for the linear transformation between the true and recovered latent spaces. The LDS error (related to the metric for Dynamical Similarity Analysis: Ostrow et al., 2023) is computed as (cf. Corollary 2):
LDS(A, = L- A
(11)
As second, more general identifiability metric for the recovered dynamics f. we introduce dynR?, which builds on Theorem lb evaluate the identifiability of non-linear dynamics. This metric computes the R² between the predicted dynamics ff and the true dynamics f. corrected for the linear transformation between the two latent spaces. Specifically, motivated by Theorem 1(b), we compute
dynR? (f. sf (f), b') +
(12)
along all time steps. Additional variants of the dynR² metric are discussed in Appendix G.
Finally, when evaluating switching linear dynamics, we compute the accuracy for assigning the correct mode at any point in time. To compute the cluster accuracy in the case of SLDS ground truth dynamics, we leverage the Hungarian algorithm to match the estimated latent variables modeling mode switches to the ground truth modes, and then proceed to compute the accuracy.
Implementation. Experiments were carried out on a compute cluster with A100 cards. On each card, we ran ~-3 experiments simultancously. Depending on the exact configuration, training time varied from 5-20min per model. The combined experiments ran for this paper comprised about 120 days of A100 compute time and we provide a breakdown in Appendix K. We will open source our benchmark suite for identifiable dynamics learning upon publication of the paper.
6 RESULTS
6.1 VERIFICATION OF THE THEORY FOR LINEAR DYNAMICS
Suitable dynamics models enable identification of latents and dynamics. For all considered classes of models, we show in Table that DCL with suitable dynamics model effectively identifies the correct dynamics. For linear dynamics (LDS), DCL reaches an R² of 99.0%, close to the oracle performance (99.5%) Most importantly, the average LDS error of our method (7.7x 10-3) i
is very close to the oracle (4.4x10 10,), in contrast to the baseline model (2. 1 x x10-1> which has substantially larger LDS error. the case of switching linear dynamics (SLDS), DCL also shows strong performance, both in terms of latent R² (99.5%) and dynamics R22 (99.9%) outperforming the respective baselines (76.8% R² and 85.5% dynamics R²). For non-linear dynamics, the baseline model fails entirely (41.0% .%). while V-SLDS dynamics be fitted with 94.1% R² for latents and 93.9% dynamics R2. We also clearly see the strength our piecewise linear approximation, as the LDS dynamics models only reaches 81. 21.2% latent identifiability and 80.3% dynamics R²
7

---

Published as a conference paper at ICLR 2025
0--
0005
0.0005
0.0050
0.0150
ground (w/s cynomiss
ours lowitching cynamics)
At
sergle 50,002 0.0005
5.0100
Figure 5: Contrastive learning of 3D non- non-linead linear dynamics following Lorenz attractor model. (a), left to right: ground truth dynamies for 10k samples with di = 0.0005 and = 0.1, estimation results for baseline (identity dynamics), DCL with V-SLDS, estimated mode sequence. (b), empirical identifiability (R²) between baseline (BAS) and V-SLDS for varying numbers discrete states K. (c. d), same layout but for 0.01 and a 0.001.
Learning noisy dynamics does not require a dynamics model. If the variance of the distribution for E, dominates the changes actually introduced by the dynamics, we find that the baseline model also able to identify the latent space underlying the system. Intuitively, the change introduced by the dynamical system is then negligible compared to the noise. In Table ("large a²), we show that recovery is possible for cases with small angles, both in the linear and non-linear case. While in some cases, this learning setup might be applicable in practice, it seems generally unrealistic to be able to perturb the system beyond the actual dynamics. As we scale the dynamics to larger values (Figure panel b and c), the estimation scheme breaks again. However, this property offers an explanation for the success of existing contrastive estimation algorithms like CEBRA-tine -time (Schneider et al., 2023) which successfully estimate dynamics in absence of dynamics model.
Symmetric encoders cannot identify non-trivial dynamics. In the more general case where the dynamics dominates the system behavior, the baseline cannot identify linear dynamics (or more complicated systems). In the general LDS and SLDS cases, the baseline fails to identify the ground truth dynamics (Table I) predicted by Corollary 1 (rows marked with X). For identity dynamics, the baseline is able to identify the latents (I12=99.56% but breaks as soon as linear dynamics are introduced (R2-73.56)).
6.2 APPROXIMATION OF NON-LINEAR DYNAMICS
Next, study in more details how the DCL identify piecewise linear or non-linear latent dynamics using the V-SLDS dynamics model.
Identification of switching dynamics. Switching dynamics depicted in Fig. 4a for four different modes of the 10 degrees dataset. DCL obtains high R² for various choices of dynamics (Fig. 4b) and additionally identifies the correct mode sequence (Fig, 4c) for all noise levels and variants of the underlying dynamics. As we increase the rotation angle used to generate the matrices, the gap between baseline and our model increases substantially.
Non-linear dynamics. Figure 5 depicts the Lorenz system as an example of non-linear dynamical system for different choices of algorithms. The ground truth dynamics vary in the ratio between dt/o and we show the full range in panels b/c. When the noise dominates the dynamics (panel a). the baseline is able to estimate also the nonlinear dynamics accurately, with 99.7%. However, as we move to lower noise cases (panel b), performance reduces to 41 1.0% Our switching dynamics model is able to estimate the system with high R² in both cases (94. 14% and 94.08%). However, note primarily succeeding at estimating the latent space, the estimated
that in this non-linear case, dynamics model did not meaningfully outperform an identity model (Appendix G).
Extensions to other distributions Pe- While Euclidean geometry is most relevant for dynamical systems in practice, and hence the focus of our theoretical and empirical investigation, contrastive learning commonly operates the hypersphere in other contexts. We provide additional results for the case a von Mises-Fisher (vMF) distribution for Pe and dot-product similarity for & in Appendix D.
8

---

Published as a conference paper at ICLR 2025
a
b
d
1.0
0.8
0,6
DN 09 poad
10
10"
10
10
10
10 10 10a 10
510 20
50
10
10
10
1.0
Custer Acc.
I
0.0
10
10" 10 10a
5100 20
10
samples
# trials
mixing_ ayers
latent dim
modes
piswitching!
Figure 6: Variations and ablations for the SLDS. We compare the V-SLDS model to the ground -tuth switching dynamics (oracle) and standard CL model without dynamics (baseline). All variations with respect to the setting with 1M time steps (1k trials and p = 0.0001 switching probability. We study the impact of the dataset size in terms of (a) samples per trial, (b) the number of trials, the impact of nonlinearity of the observations in terms of (c) number of mixing layers, the impact of complexity of the latent dynamics in terms of (d) latent dimensionality, (e) number of modes switch in between and (f) switching frequency paramameterized via the switching probability.
samples), L = 4 mixing layers, = latent dimensionality, 5 modes,
6.3 ABLATION STUDIES
For practitioners leveraging contrastive learning for statistical analysis, it is important to know the trade-offs empirical performance in relation to various parameters. In real- experiments, the most important factors are the size of the dataset, the trial-structure of the dataset, the latent dimensionality we can expect to recover, and the degree of non-linearity between latents and observ- ables. We consider these factors of influence: As reference, we use the SLDSS system with 6D latent space, 1M samples (1k trials x 1 samples), matrices, 65,536 negative samples per batch; batch size 2,048, and learning rate 10-3
mixing layers, 10 degrees for the rotation
Impact of dataset size (Fig, 6a). We keep the number of trials fixed to 1k. As we vary the sample size per trial, R² degrades for smaller dataset, and for the given setting we need at least 100 points per trial to attain identifiability empirically. We outperform the baseline model all cases.
Impact of trials (Fig. 6b). We next simulate fixed number of 1M datapoints, which split into trials of varying length. We consider 1k, 10k, 100k, and IM as trial lengths. Performance is stable for the different settings, even for cases with small trial length (and less observed switching points). DCL consistently outperforms the baseline algorithm and attains stable performance close to the theoretical maximum given by the ground-truth dynamics.
Impact of non-linear mixing (Fig. 6c). All main experiments have been conducted with L, = mixing layers in the mixing function g. Performance of DCL stays at the theoretical maximum as we in- crease the number of mixing layers. As we move beyond four layers, both oracle performance in R² and our model declines, hinting that either (1) more data or (2) a larger model required to recover the dynamics successfully in these cases.
00000
0.01
Impact of dimensionality (Fig, 6d). Increasing latent dimension- ality does not meaningfully impact performance of our model. We found that for higher dimensions, it is crucial use large number of negative examples (65k) for successful training.
0.8
10"
modes
Figure 7: Impact of modes for non-lincar dynamics in the Lorenz system for different sys- tem noise levels a, averaged over dt.
Number of modes for switching linear dynamics fitting (Fig, 6e). Increasing the number of modes in the dataset leads to more suc- cessful fitting of the R² for the baseline model, but to a decline accuracy. This might be due to the increased variance: While this helps the model to identify the latent space (dynamics appear more like noise), it still fails to identify the underlying dynamics model, unlike DCL which attains high R²2 and cluster accuracy throughout.

---

Published as a conference paper at ICLR 2025
Robustness to changes in switching probability (Fig. 6f). Finally, we vary the switching probability. Higher switching probability causes shorter modes, which are harder to fit by the V-SLDS dynamics model. Our model obtains high empirical identifiability throughout the experiment, but the accuracy metric begins to decline when = 0.1 and p 0.2.
Number of modes for non-linear dynamics fitting (Fig. 7). We study the effect of increasing the number of matrices in the parameter bank W in the V-SLDS model. The figure depicts the impact of increasing the number of modes for DCL on the non-linear Lorenz dataset. We observe that increasing modes to 200 improves performance, but eventually converges to a stable maximum for all noise levels.
DISCUSSION
The DCL framework is versatile and allows to study the performance of contrastive learning in conjunction with different dynamics models. By exploring various special cases (identity, linear, switching linear), our study categorizes different forms of contrastive learning and makes predictions about their behavior in practice. In comparison to contrastive predictive coding (CPC; Oord et al., 2018) or wav2vec (Schneider et al., 2019), DCL generalizes the concept of training contrastive learning models with (explicit) dynamics models. CPC uses RNN encoder followed by linear projection, while wav2vec leverages CNNs dynamics models and affine projections. Theorem 1 applies to both these models, and offers explanation for their successful empirical performance.
Nonlinear ICA methods, such as TCL (Hyvarinen & Morioka, 2016) and PCL (Hyvarinen & Morioka, 2017) provide identifiability of the latent variables leveraging temporal structure of the data. Com- pared to DCL,, they do not explicitly model dynamics and assume either stationarity or non-stationarity of the time series (Hyvärinen et al., 2023), whereas DCL assumes bijective latent dynamics, and focuses on explicit dynamics modeling beyond solving the demixing problem.
For applications in scientific data analysis, CEBRA (Schneider al.. 2023) uses supervised or self-supervised contrastive learning, either with symmetric encoders or asymmetric encoder functions. While our results show that such an algorithm able to identify dynamics for a sufficient amount of system noise, adding dynamics models is required as the system dynamics dominate. Hence, the DCL. approach with LDS or V-SLDS dynamics generalises the self-supervised mode of CEBRA and makes it applicable for broader class of problems.
Finally, there is connection to the joint embedding predictive architecture (JEPA; LeCun, 2022; Assran et al., 2023). The architecture setup of DCL can be regarded as special case of JEPA, but with symmetric encoders to leverage distillation of the system dynamics into the predictor (the dynamics model). In contrast to JEPA, the use of symmetric encoders requires contrastive loss for avoiding collapse and, more importantly, serves as the foundation for our theoretical result.
A limitation of the present study is its main focus simulated data which clearly corroborates theory but does not yet demonstrate real-world applicability. However, our simulated data bears the signatures of real-world datasets (multi-trial structures, varying degrees of dimensionality, number of modes, and different forms of dynamics). A challenge is the availability of real-world benchmark datasets for dynamics identification. We believe that rigorous evaluation of different estimation methods on such datasets will continue to show the promise of contrastive learning for dynamics identification. Integrating recent benchmarks like DynaDojo (Bhamidipaty al., 2023) or datasets from Chen et al. (2021) with realistic mixing functions (g) offers a promising direction for evaluating latent dynamics models. As demonstration of real-world applicability, we benchmarked DCL on a neural recordings dataset in Appendix J.
8 CONCLUSION
We proposed first identifiable, end-to-end, non-generative inference algorithm for latent switching dynamics along with an empirically successful parameterization of non-linear dynamics. Our results point towards the empirical effectiveness of contrastive learning across time-series, and back these empirical successes by theory. We show empirical identifiability with limited data for linear, switching linear and non-line -ineer dynamics. Our results add to the understanding of SSL's empirical success, will guide the design of future contrastive learning algorithms and most importantly, make SSL amenable for computational statistics and data analysis.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
caleimaee--eeeeeeeeeeeee Code. Code is available at ht tttps:: gitbbb. -or. Apache 2.0 license. Experimental and implementation details for the main text are given in section and for each experiment of the Appendix within the respective chapter.
under an
Theory. Our theoretical claims are backed by complete proof attached in Appendix A. Assumptions are outlined in the main text (Section 3) and again in more detail Appendix
Datasets. We evaluate our experiments on variety of synthetic datasets. The datasets comprise different dynamical systems, from linear to nonlinear.
Compute. Moderate compute resources are required to reproduce this paper. As stated in section 5, we used around 120 days of GPU compute on A100 to produce the results presented in the paper. We provide more detailed breakdown in Appendix K.
AUTHOR CONTRIBUTIONS
RGL and TS: Methodology, Software, Investigation, Writing-Editing. SiS: Conceptualization, Methodology, Formal Analysis, Writing-Original Draft and Writing-Editing.
ACKNOWLEDGMENTS
We thank Luisa Eck and Stephen Jiang for discussions on the theory, and Lilly May for input on paper figures. We thank the five anonymous reviewers at ICLR for their valuable and constructive comments on our manuscript. This work was supported by the Helmholtz Association's 's Initiative and Networking Fund on the HAICORE@ KIT and HAICORE @ FZJ partitions.
REFERENCES
Guy Ackerson and Fu. On state estimation in switching environments. IEEE transactions on automatic control, 15(1):10-17, 1970.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Sepe-super learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1281 1312. PMLR, 2022.
Carles Balsells-Rodae Yixin Wang, and Yingzhen Li. On the identifiability of switching dynamical systems. arXiv preprint UrXiv:2305.15925. 2023.
Ror Bellman and Karl Johan Aström. On structural identifiability. Mathematical biosciences, 7(3-4):
329-339, 1970.
Logan Mondal Bhamidipaty, Tommy Bruzzese, Caryn Tran, Rami Ratl Mrad, and Max Kanwal. Dynadojo: an extensible benchmarking platform for scalable dynamical system identification. In Thirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0723 2021.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv: 14165, 2020.
Steven L. Brunton, Joshua I Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-3933 2016.
Chaw-Bing Chang and Michael Athans. State estimation for discrete systems with switching parameters. IEEE Transactions Aerospace and Electronic Systems, (3):418-425 1978.
11

---

Published as a conference paper at ICLR 2025
Boyuan Chen, Kuang Huang, Sumand Raghupathi, Ishaan Chandratreya, Qiang Du, and Hod Lipson. Discovering State Variables Hidden in Experimental Data, December 2021. URL http://
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordinary differential equations. arXiv preprint arXiv:2011.03902, 2020a.
Sheng Chen and Steve Billings. Representations of non-linear systems: the narmax model. International journal of control, 49(3):1013-10522 1989.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Silvia Chiappa al. Explicit-duration markov switching models. Foundations and Trends@ in Machine Learning, 7(6):803-88) 886, 2014.
Sy-Miin Chow and Guangjian Zhang. Nonlinear regime- state-space (rsss) models. Psy- chometrika, 78:740-768. 2013.
Hanjun Dai, Bo Dai. Yan-Ming Zhang, Shuang Li, and Le Song. Recurrent hidden cemi-markov model. In International Conference on Learning Representations, 2022.
Stéphane d' Ascoli, Sören Becker, Alexander Mathis, Philippe Schwaller, and Niki Kilbertus. Odeformer: Symbolic regression of dynamical systems with transformers. arXiv preprint arXiv:2310.05573, 2023.
Saskia EJ de Vries, Jerome Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-151, 2020.
Zhe Dong, Bryan Seybold, Kevin Murphy, and Hung Bui. Collapsed amortized variational inference for switching nonlinear dynamical systems. In International Conference on Machine Learning, pp. 2638-2647, PMLR, 2020.
Lea Duncker, Gergo Bohner, Julien Boussard, and Maneesh Sahani. Learning interpretable continuous-time models of latent stochastic dynamical systems. In International conference on machine learning, pp. 1726-1734. PMLR, 2019.
Jeffrey I. Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
Yuanjun Gao, Evan W Archer, Liam Paninski, and John P Cunningham. Linear dynamical neural population models through nonlinear embeddings. Advances in neural information processing systems, 29, 2016.
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning. arXiv preprins arXiv:2403.00504, 2024.
Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural computation, 1244)8811-844, 2000.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00752. 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arX0v:2111.00396, 2021.
David Ha and Jürgen Schmidhuber. World models. CoRR, abs/1803.10122. 2018. URL http:
//arxiv. orgo abw/aas//000 wwwg/bag//laaeeeo
10122.
Hermanni Hälvä, Sylvain Le Corff, Lue Lehéricy, Jonathan So. Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34: 14:1624-1633, 2021.
12

---

Published as a conference paper at ICLR 2025
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, Pp. 4182-4192. PMLR, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). 1rX6. 08415, 2016.
ar preprint
Cole Hurwitz, Nina Kudryashova, Arno Onken, and Matthias H Hennig. Building population models for large-scale neural recordings: Opportunities and pitfalls. Current opinion in neurobiology, 70: 64-73, 2021.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastivve learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent compo- nent analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10):
100844, October 2023, ISSN 26663899, doi: 10. 00.10016i.paater2233..00444 URL https:
//linkinghub. e elsevier com/ ceteretrriie
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint 161100 144, 2016.
-lexan-Ae Kamienny, Stéphane d' Ascoll, Guillaume Lample, and François Charton. End-to- end symbolic regression with transformers. Advances in Neural Information Processing Systems, 35:10269-102811 2022.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2022-06-27. Open Review, 62(1):1-62, 2022.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial intelligence and statistics, pp. 914-922 PMLR, 2017.
Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. 13557-13603. PMLR, 2022.
Stefan Matthes, Zhiwei Han, and Hao Shen. Towards unified framework of contrastive learning for disentangled representations. Advances in Neural Information Processing Systems, 36:67459
67470, 2023.
Leonard McGee and Stanley F Schmidt. Discovery of the kalman filter as practical tool for aerospace and industry. Technical report, 1985.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. anXiv preprint arXiv:1807.037488 2018.
Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and Ila Fiete. Beyond geometry: Comparing the temporal structure of computation neural circuits with dynamical similarity analysis, 2023. URL 101 68.
https://arae wwwpwwwwwwwwwwooo/ooot/oo.ooorrp
13

---

Published as a conference paper at ICLR 2025
Chethan Pandarinath, Daniel J O'Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-815, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations.
In International Conference on Machine Learning, pp. 9030-9039, PMLR, 2021.
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S. Zimmermann, and Wieland Brendel. Infonce: Identifying the gap between theory and practice. 2024. https:
//arxivv... org. /00b////407 00143.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint 1rXiv:194..05860 2019.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):300-363 2023.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Timeecontrastive networks: Sepe-supe learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Ruian Shi and Quaid Morris. Segmenting hybrid trajectories using latent odes. In International Conference on Machine Learning, pp. 9569-9579. PMLR, 2021.
Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with jacobian switching linear dynamical systems. Advances Neural Information Processing Systems, 11116600-167711 2021.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-flov networks (gin). arXiv preprint arXiv:2001.0487. 2020.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Leurning, pp- 9929-9939. PMLR. 2020.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint 1rXiv:21 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. Advances in Neural Information Processing Systems, 35:26492 26503,, 2022.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12979-12990. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
Supplementary Material
A Proof of the main result
16
Kernel density estimate correction
C Additional Related Work
21
D Von Mises-Fisher (vMF) conditional distributions
23
E Additional plots for SLDS
24
F Generalization Train- vs. Test Set
25
G Variations and additional baselines for the Dyn Metric
27
G.I Method
27
G.2 Results
27
H Non-Injective Mixing Functions
28
H.1 Experimental Validation
28
H.2 Results
29
Dynamical Systems with Control Signal
31
1.1
Empirical Verification
31
1.2
Experiment Details
31
1.3
Results
32
J Application to Real-World Data
33
J.1 Methods
................................
33
J.2 Results
................................
34
J.3 Discussion
34
K Computational Requirements
35
L On component-wise vs. linear identifiability
36
Comparison additional time-series models
37
M.1 Verifying Baselines
M.2 Experiment Details
37
M.3 Results..
38
15

---

Published as a conference paper at ICLR 2025
A PROOF OF THE MAIN RESULT
We re-state Theorem from the main paper, and provide a full proof below:
Theorem (Contrastive estimation of non-linear dynamics). Assune that
(AI) A time-series dataset {ye}e= is generated according to the ground-truth dynamical system in Eq. 5 with bijective dynamics model and injective mixing function g-
(A2) The system noise follows an iid normal distribution, p(E,) = =,, Ee))
(A3) The model vo is composed of an encoder h, dynamics model f. correction term and the similarity metric o(u, v) = --1 ---) o1I2 and attains the global minimizer of Eq. 3.
Then, in the limit of - oo for any point x: in the support of the data marginal distribution:
(a) The composition of mixing and de-mixing h(g((=)) Lx + a bijective affine transform. and L QE. 1/2 with unknown orthogonal transform Q E Tg and offset b E Rd.
(b) The estimated dynamics f are bijective and identify the true dynamics f up the relation f(x) = (z= (x)b) b.
Proof. Our proof proceeds in three steps: First, we leverage existing theory (Wang & Isola, 2020; Zimmermann al., 2021) to arrive the minimizer of the contrastive loss, and relate the limited sample loss function to the asymptotic case. Second, we derive the statement about achieving successful demixing in Theorem I(a). Finally, we derive the statement in Theorem 1(a) about structural identifiability of the dynamics model.
Step 1: Minimizer of the InfoNCE loss. By the assumption e, normally distributed, we obtain the positive sample conditional distribution
p(a++1 (..) (N (xx), Ee).
(13)
The negative sample distribution g ((xx) obtained by sampling uniformly from all time steps and can hence be written as a Gaussian mixture along the dynamics imposed by f.
((x)) T
Σ Pe - x..))
(14)
T N(x - xx)f(xx-1).
Σ
(15)
T
We use these definitions of and to study the asymptotic case of our loss function. For T + co. due to Wang & Isola (2020) we can rewrite the limit of our loss function (Eq. 3)
[
lim Ty, log (2 N)) log
T T+x
f 9(v)
[-/
p(y (olll))(( (o..yyyydde ddy - log / q(3') exply(y y/ d')]
(16)
It was shown (Proposition 1, Schneider et al., 2023) that this loss function is convex in e with the unique minimizer
p(x' (x) ((x')
(g(p)) = log
+ c(w)
(17)
where e: IRd 1 R. an arbitrary scalar-vall valued function. Note that we also expressed y = (ar)), y (xx') to continue the proof in terms of the relation between original and estimated latents. We insert the definition of the model on the left hand side. Let also denote h g =: r. and the definition of the ground-truth generating process the right hand side to obtain
((((r())), a(x') = log p(= [x()) log q(x') + c(x).
(18)
Inserting the potential? as a(x) log ((x) simplifies the equation to d(f(r(x))) r(x')) log (f)))) c(x)
(19)
16

---

Published as a conference paper at ICLR 2025
From here onwards, will use that is the negative squared Euclidean norm (A.3) and correspond-
ingly, the positive conditional a normal distribution with concentration A = Σn (A.2),
- -I]f(r(z)) r = -f')
x') + (zx)
(20)
where we pulled the normalization constant of p into the function c' for brevity.
Step 2: Properties of the feature encoder. Starting from the last equation, we compute the derivative with respect to æ and x' on both sides and obtain
(a' J - (z)
(21)
Because this equation holds for any a' E supp q independently of æ, we can conclude that the Jacobian matrix of needs to be constant. From there it follows that is affine. Let us write r(x) = Læ + b. Then, the Jacobian matrix J, L. Inserting this yields
L' [L b) (z)
(22)
We next establish that L has full rank: because the dynamies function f bijective by assumption, J (a:) has full rank d. A has full rank by assumption about the distribution P=(E). All matrices on the LHS are square and need to have full rank as well for any point æ. Hence, we can conclude that L has full rank, and likewise Jg. From there, we conclude that r and f are bijective.
Next, we derive additional constraints on the matrix L. We insert the solution for r obtained so far in Eq. 20,
l[f b) La' b11 (f(x) - x') A(f(z) x²) cx(x)
(23)
and take the derivative twice with respect to x', to obtain
L
AL
(24)
Without loss of generality, we introduce Q E Rdxd to write L in terms of A as L = QA 2/2 Inserting into the previous equation lets us conclude
- Q Q/ = QQQ - - 1/2
(25)
-1/22 - II
(26)
from which follows that Q is orthogonal matrix. Hence, L is composition of orthogonal transform and Σg 1/2 concluding the first part of the proof for statement (a).
Step 3: Dynamics. To derive part (b), start at the condition Eq. 20 again to determine the value of
c((x). We can consider two special cases:
f(x) = x': c(x) = -1f(((()))) - -(())
(27)
= ( d(x) (f(z) x²) A x²) 20
(28)
where we use that the concentration matrix A of a Normal distribution is positive semi-definite When combining both conditions for points where f(x) = x' and f(r(z)) = r(x') the only admissible solution is c'xx) 0 for points with f(r(x) r rff())), i.c f(x)
(x))), hinting the final solution. However, we have not shown yet that this solution is unique.
To show uniqueness, without loss of generality, we use the ansatz (with residual v)
f(x) Anf(La: b) + d1 v(x),
(29)
and computing the derivative with respect to x yields
( A A, J La: + b) Ja(a:...
(30)
Thhi is feasible in practice by parameterizing (x(zz) as kernel density estimate, but empirically often not required. See Appendix B for additional technical details.
17

---

Published as a conference paper at ICLR 2025
We insert this into Eq. 22 and obtain
L' J1(LE + b) AAJ / b) + (J,(z)
(31)
(LT AAI) , (La -)) - (J)
(32)
AA ( +b)
(33)
The left hand side is a constant, hence the same needs to hold true for the right hand side. Without loss of generality, let introduce arbitrary matrix A2 set as this constant,
Jo(x) (Læ + b) = A
(34)
Ju(z) J j (Lx b)
(35)
which only admits the solution
v(x) A2f(Lar b) + d2,
(36)
where we introduced an additional integration constant d2. Inserting this into the ansatz in Eq. 29 gives
f(z) A.f(L 6) + v(x),
(37)
f(x) = (A1 + A2) + b) + (d1 + d2).
(38)
Using the shorthand A Ai A2, d di + d2 we can repeat the steps in Eqs. 30-33 to arrive at the condition
L - ( b- =0 0.
(39)
Since all matrices have full rank, the only valid solution is A = A = -- Inserting back into the ansatz yields the refined solution
f(x) L f- La + b) d,
(40)
and for brevity, we let & = f(r(x)) = f(Lx b):
f(x) = - d.
(41)
We then insert the current solution into Eq. 20 and input r which gives
1E Lx' b112 = (L- + d x²) A(L & d d'x') + (x)
(42)
(L 1L-'E + -') T²) L LL d - x') (x)
(43)
= [ Ld Lx' + d (xx
-
(44)
d(x) || Lx' -bl || Læ Ld
(45)
Let us denote u - Lx' and note that v and æ remain independent variables. We then get
c(xx) |/2 b11 - ||0 + 2d| - (b Ld) + |bb|| Ld]
(46)
(47)
Because v and æ vary independently and the equation true for any pair of these points, both sides of the equation need be independent of their respective variables. This is true only b Ld. Hence, it follows that
d(xx) 0 and - -. Inserting d into Eq. 40 gives the final solution,
(48)
L- f(Lx + bb) - L-16.
(49)
Solving for f gives
f(La + b) Lf(x) +b
(50)
f(x) = Lf(L 'x b)) + b= (rofor
(51)
which concludes the proof.
18

---

Published as a conference paper at ICLR 2025
B KERNEL DENSITY ESTIMATE CORRECTION
Theorem requires to include "potential function" a into our model. In this section, we discuss how this function can be approximated by kernel density estimate (KDE) in practice. The KDE intuitively corrects for the case of non-uniform marginal distributions. Correcting with a overcomes the limitation requiring uniform marginal distribution discussed before (Zimmermann al., 2021). While other solutions have been discussed, such as training separate MLP (Matthes et al., 2023), the KDE solution discussed below is conceptually simpler and non-parametric.
For the models considered in the main paper, we considered representation learning in Fuclidean space, while Appendix D contains some additional experiments for the very common case training embeddings the hypersphere. For both cases, we can parameterize appropriate KDEs.
For the Euclidean case, we use the KDE based on the squared Euclidean norm,
M
((x)
Σ exp
x x;
x; ~ q(x).
(52)
e.M
We note that in the limit ((æ) - q()). This is also the case used in Theorem However, this estimate depends the ground truth latents 24, which are not accessible during training. Hence, we need to find an expression that depends the observable data. We leverage the feature encoder h to express the estimator as
0, .M -> 00, this estimate converges to the correct distribution,
Σ exp (- (- cM
1
[lh(()) hb(x))1
(n((v)
Yi ((y)-
(53)
We express this estimator in terms of the final solution, r(x) (() QΣ b in the theorem. If we express the solution in terms of the ground truth latents again, the orthogonal matrix Q vanishes and we obtain
Eu 1/2
x.)|
qn (x)
Σ
exp
y: (ly)
(54)
cM
This corresponds to KDE using Mahalanobis distance with covariance matrix E., which is valid KDE of
We can derive similar argument when computing embeddings and dynamics on the hypersphere. When, a von Mises-Fisher distribution suitable to express the KDE, and we obtain
(p(N) ((x: = M Σ
M
exp(xx T as), 0; ~ g(x)
(55)
where Cpp (1:) ( is the normalization constant of the von Mises-Fisher distribution. This again approaches the correct data distribution for + q M, + oc. Following the same arguments above, but
1.0
Ko
Kp
Ks
0.9
KDE K 1000
KDE < 1000
KDE k 10,000
0.8
300
300
100 wio KDE
w/o KDE
wio KDE
2-3
21
25
2-3
2
25
2
23
26
Figure 8: Introducing KDE into the loss allows to compensate for non-unifome marginal distribution. We show performance in terms of R² across datasets with incrcasingly non-uniform marginal. We replicate the data geaa--enrraiinnggg process and experimental setup performed by Zimmermann et (2021, Figure
19

---

Published as a conference paper at ICLR 2025
using r(z) h(g(x)) = Qu the indeterminacy the hypersphere, express this in terms of the ground truth latents,
Cp(K))
M
án (x) =
Σ
exp
(xT(2)) rrx;))
(56)
N
t=1
Cp(())
M
exp
(RT x.).
(57)
N
which is again valid KDE.
is interesting to consider the effect of the KDE on the loss function. Inserting & t v log q into the loss function yields
logg (x|x=, ,N) -16((,)) xt;) log ((x;))) )) + log
(an ) log ((m5),
(58)
i=1
-,,,, x:) + log ((x; log
(59)
g(x0g)
N
In (xx
--b(w xxt) + log
(60)
on (x)
iml
-,i,; + log
wh (x 2 e (a
(61)
with the importance weights wn, (ar," .z))
Intuitively, this means that the negative examples are re-whithted according to the density ratio between the current positive and each negative sample.
On (an
(z,)))
Empirical motivation. Figure shows preliminary results on applying this KDE correction to contrastive learning models. We followed the setting from Zimmermann et al. (2021) and re-produced the experiment reported in Fig. their paper. We use 3D latents, a 4-layer MLP as non-linear mixing function with final projection layer 50D observed data. The reference, positive and negative distributions are all vMF' parameterized according to x (x-axis) in the case of the reference and negative distribution and Ki for the positive distribution.
The grey curve shows the decline in empirical identifiability (R²) the uniformity assumption is violated by an increasing concentration y (X-axis). Applying a KDE correction to the data resulted in substantially improved performance (red lines).
However, when testing the method directly on the dynamical systems considered in the paper, we did not found substantial improvement performance. One hypothesis for this is that the distribution of points on the data manifold (not necessarily the whole IR is already sufficiently uniform. Hence, while the theory requires inclusion of the KDE term (and did not degrade results), we suggest to drop this computationally expensive term when applying the method on real-world datasets that are approximately uniform.
20

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RELATED WORK
Contrastive learning, An influential and conceptual motivation for our work is Contrastive Predictive Coding (CPC) (Oord et al.. 2018) which uses the InfoNCE loss with an additional non non-linear projection head implemented as an RNN to aggregate information from multiple time steps. Then, an affine projection is used for multiple forward prediction steps. However, contrary to our approach, the "dynamies model" is not explicitly parameterized, limiting its interpretability. Similar frameworks have been successfully applied across various domains, including audio, vision, and language, giving rise to applications such as wav2vec (Schneider et al., 2019), time contrastive networks for video (TCN; Sermanet et al., 2018) CPCv2 (Henaff, 2020). An interesting parallel to DLC+LDS in image representation learning is AnInfoNCE (Rusak et al., 2024) which extends a SimCLR- (Chen et al., 2020b) training setup by additional learnable diagonal matrix in its similarity function.
Non-Contrastive learning. Models such as data2vec (Baevski et al., 2022) and JEPA (Assran et al., 2023) learns a representation by trying to predict missing information in latent space, using MSE loss. JEPA uses asymmetric encoders, and on top a predictor model in latent space parameterized by a neural net. However, these approaches do not provide any identifiability guarantees.
System identification. In system identification, a problem closely related to the one addressed in this work is known as "nonlinear system identification. Widely used algorithms for this problem include Extended Kalman Filter (EKM) (McGee & Schmidt, 1985) and Nonlinear Autoregressive Moving Average with Exogenous inputs (NARMAX) (Chen Billings, 1989). EKF based on linearizing g and f using first-order Taylor-series approximation and then apply the Kalman Filter (KF) to the linearized functions. NARMAX, on the other hand, typically employs a power-form polynomial representation to model the non-linearities. In neuroscience, practical (generative algorithms) include systems modeling linear dynamics (fLDS; Gao et al., 2016) or non-linear dynamics modelled by RNNs (LFADS; Pandarinath et al., 2018). Hurwitz et al. (2021) provide detailed summary of additional algorithms.
Nonlinear ICA. The field of Nonlinear ICA has recently provided identifiability results for identifying latent variables, usually employing auxiliary variables such as class labels or time information (Hyvarinen & Morioka, 2016; 2017: Hyvarinen al., 2019; Khemakhem et al., 2020; Sorrenson et al., 2020). In the case of time series data, Time Contrastive Learning (TCL) (Hyvarinen & Morioka, 2016) uses a contrastive loss to predict the segment-ID of multivariate time-series which was shown to perform Non-linear ICA. Permutation Contrastive Learning (PCL) (Hyvarinen & Morioka, 2017) permutes the time series and aims to distinguish positive and negative pairs.
Temporal causal representation learning. In Nonlinear ICA, the factors are assumed to be in- dependent, subject to some indeterminacy in the original latent variables. However, this approach encounters challenges when the latent variables have time-delayed causal relationships. Approaches like LEAP (Sorrenson et al., 2020) and TDLR (Yao et al., 2021) address these challenges in both stationary and non-stationary environments, even when the transition function's 's parametric form is unknown. CaRING (Yao et al., 2022) extends these results to cases where the mixing function is non-invertibe inniineertible. Lastly, CITRIS (Lippe et al., 2022) introduces intervention target information to enhance the identification of latent causal factors. In this work, we do not aim to estimate the temporal causal graph. Instead, we focus estimating the dynamics model f using interpretable and explicitly parameterized dynamics model (e.g. V-SLDS) which can later be analyzed for applications such as scientific discovery.
Switching Linear Dynamical Systems. Several papers propose methods to infer SLDSs (Ackerson & Fu, 1970; Chang& & Athans, 1978; Ghahramani & Hinton, 2000), leading to variety of extensions and variants. For example, Recurrent SLDSs (Linderman et al., 2017; Dai et al., 2022) address state- dependent switching by changing the switch transition distribution to (y: 3/- 1)) allowing for more flexible dependencies on previous states. Another extension, Explicit duration SLDS introduces additional latent variables to model the distribution of switch durations explicitly (Chiappa et al., 2014). Some approaches relax the assumption of linear dynamics, such as in the case of SNLDS and RSSSM, where the dynamics model is assumed to be nonlinear (Dong et al., 2020; Chow & Zhang, 2013). In the context of Nonlinear Independent Component Analysis (ICA), recent extensions include structured data generating processes (e.g., SNICA; Hälvä et al., 2021) which were shown to be useful for the inference of switching dynamics. In this vein, Balsells-Rodas et al. (2023) proposed additional identifiability theory for the switching case. Other approaches, based on Neural Ordinary Differential Equations (Neural ODEs; Chen al., 2020a; Shi & Morris, 2021), or methods aimed at
21

---

Published as a conference paper at ICLR 2025
discovering switching dynamics within recurrent neural networks (Smith et al., 2021), also present interesting avenues for modeling switching dynamics.
Deep state-space models. Recently, (deep) state-space models (SSMs) such S4 Mamba (Gu et al., 2021; Gu Dao, 2023) have emerged as a promising architecture. These models are particularly well-suited for capturing long-range dependencies, making them attractive choice for sequence modeling tasks.
Symbolic Regression. An alternative approach to modeling dynamical systems is the use of symbolic regression, which aims to directly infer explicit symbolic mathematical expressions governing the underlying dynamical laws. Examples include Sparse Identification of Nonlinear Dynamics (SINdy; Brunton et al., 2016), as well as more recent transformer- based models (Kamienny et al., 2022: Ascoli al., 2023), which have demonstrated promise discovering interpretable representations of dynamical systems.
22

---

Published as a conference paper at ICLR 2025
D VON MISES-FISHER (VMF) CONDITIONAL DISTRIBUTIONS
In the main paper, we have shown experimental results that verify Theorem 1 in the case of Normal distributed positive conditional distribution and using the Euclidean distance. This approach has allowed for modeling latents and their dynamics in Euclidean space, which we argue is the most prac- tical setting to apply DCL in. However, self-supervised learning methods and especially contrastive learning have commonly been applied to produce representations on the hypersphere and using the dot-prodt dot-product distance (Oord et al., 2018; Schneider et al., 2023; Wang & Isola, 2020; Zimmermann et al., 2021; Chen et al., 2020b).
Here we validate empirically that Theorem equally holds under the assumption of vMF conditional distributions and using the dot-product distance o(æ, y) x: y part of the loss. We exper- iments as in Table for the case where the true dynamics model f is a linear dynamical system. Additionally, we vary the setting similar to Figure 4 to show increasing At (angular velocity).
We compare:
DCL (ours) - with linear dynamics: f(z) Az.
GTD the ground-truth dynamics model (LDS) f(x) = Ax.
No dynamics the baseline setting use throughout the paper f(x) x.
Asymmetric a variation on the baseline setting that uses asymmetric encoders (one for reference, one for positive or negative) which would be a possible fix of Corollary 1. We can obtain this setting by skipping the explicit dynamics modeling, and defining two encoders h1, h2 which relate as follows: h f = h1, h h2.
1.0
GTD / DynCL
1.0
/DmOLL CommCL/Aspa Asymetric
Asymetric
0.8
0.8
R²
0.6
R²
No dynamics
0.6
0.4
0.01
No dynamics
0.4
0.2
012345
10
20
012345
10
20
Rotation Angle (Degrees)
Rotation Angle (Degrees)
Figure 9: Our findings from Table hold equally under vMF noise distribution when using LDS ground truth dynamics. We show empirical identifiability of the latents in terms of It2 under varying a) angles of the rotation dynamics i.c. angular velocity At (x-axis) and b) the magnitude of the dynamics noise (pancls, left: low noise, right: high noise)
Similar to our results for the Euclidean case (Table 1), in Figure 9 we show results that experimentally verify Theorem for latent dynamics on hypersphere and using vMF as conditional distribution. Both for low (left panel) and high (right panel) variance of the conditional distribution we can see that DCL effectively identifies the ground truth latents on par with the oracle (GTD) model performance. On the other hand, the baseline, standard time contrastive learning without dynamics, can not identify ground truth latents with underlying linear dynamies as predicted by Corollary 1. This prediction is only violated in the case where the variance of the noise distribution is high enough, such that the noise dominates the changes introduced by the actual dynamics. This is the case for dynamics with rotations up to 4 degrees for 0.01 and angles up to 10 degrees for 0.1.
23

---

Published as a conference paper at ICLR 2025
E ADDITIONAL PLOTS FOR SLDS 0 = 0.0001
0 0.01
Figure 10: Visualizations of 6D linear dynamical systems at 0.0001 (left) and a 0.01 for 10 degree rotations. These systems used in our SLDS experiments.
24

---

Published as a conference paper at ICLR 2025
F GENERALIZATION TRAIN- vs. TEST SET
In the main paper, all metrics are evaluated using the full training dataset of the respective experiment. We argue that this sufficient for showing the efficacy of our model and verifying claims from the theory in section 3 because a) in self-supervised learning, the model learns generalizable representa- tions through pretext tasks, making overfitting less of a concern; b) the metrics we are interested in are about uncovering the true underlying latent representation and dynamics of the available training data, not new data; and c) most importantly, we ensured that the training dataset is large enough to approximate the full data distribution.
Nonetheless, here we show series control experiments to re-evaluate models on new and unseen data. We do by following the same data generating process of the given experiment (same dynamics model and mixing function) and sample completely new trials (10% of the number of trials of the training dataset). Every new trial starts random starting point, with randomly sampled new mode sequence and regenerated with different seeds for the dynamics noise.
First, re-evaluated every experiment of Figure 6 on the test dataset generated described above and show these results in Figure 11. Comparing those results to the train dataset version of Figure shows that there is almost no difference in the performance (with regard identifiability and systems identification). The difference are so small, that visually comparing the results almost becomes impractical, so we additionally provide the exact numbers of the first panel (variations on the number of samples per trial) in Table 2.
Finally, qualitatively and quantitatively show the difference between the train and test datasets we provide a) depictions of the ground truth latents of 5 random test trials and their closest possible matching trial from the training set in Figure 12 and distribution of the distances (in terms of R² between the data from the test and train trials) between all test trials of one of random test set and their closest trial from the training dataset in Figure 13.
0.6
10
10a
10a
10a
10
102
102
10s
10
6
510 20
50
10-1
10-2
10-
1.0
Cluster ACC
10:
10a
10"
10
10.
10a
10a
10
10a
6 mixing_laaye layers
6
510 20
50
10-
10 10
latent
modes
piswitching)
Figure 11: Same as Figure (Variations and ablations for SLDS), but re-evaluated on a newly generated test data with different starting points.
Table detailed view on the #samples panel from Figure 6 and 11 showing the difference between train and test set evaluation.
DCL (ours) Rº (train)
CL w/o dynamics R² (train)
CL w/ ground truth dynamics
R (test)
R² (test)
R? (train R (test)
samples
10
0.991 0.00137
0.989 + 0.00172
0.923 + 0.03703
0.917 + 0.04000
0.954
0.00399
0.952 + 0.00442
100
0.995 1: 0.00106
0.994
1: 0.00108
0.786 1 0.06794
0.791 1 0.06931
0.990 = 0.00107
0.990 :: 0.00099
0.00074
0.995 0.00078
0.06671
0.06973
0.990
0.990
10000
0.996
: 0.00046
0.996 :: 0.00044
0.694 1 0.06937
0.801 +: 0.10568
0.991 = 0.00509
0,991 :: 0.00425
25

---

Published as a conference paper at ICLR 2025
Trial (D17)
Trial (8
(RR
Trial (R? 0.73)
Trial 0.80)
Tot Peforence Bioh Icheeet matc?)
Figure 12: Ground truth latents from five random trials of the testsets for Figure 11 and their closest match within the conresponding trainset. The closest match is evaluated by computing the R"-Score between given trial from the testset and every possible trial.
0.4
02
0.0
0.2
0.4
0.6
0.8
100
Score
Figure 13: Histogram of all RR-Scores between every trial from the testset and its closest possible match from the trainset shown in Figure 12.
26

---

Published as a conference paper at ICLR 2025
G VARIATIONS AND ADDITIONAL BASELINES FOR THE DYN METRIC G.1
METHOD
As an addition to Table 1, we analyse the DynR² in more detail. In Table we show variants for the metric. Firstly, we modify the number of forward prediction steps,
(x) = of.... (x)
(62)
1I times
and respectively for f in relation to f. We then consider two variants of Eq. 12. Firstly, we perform multiple forward predictions (n > 1) and compare the resulting embeddings:
r2_score(f (t), Lf" LL 6') + b).
(63)
A rationale for this metric is that the prediction task becomes increasingly difficult with an increasing number time steps, and errors accumulate faster.
Secondly, as an additional control, we replace f with the identity, and compute
12_score( (L' +
(64)
This metric can be regarded as a naivve baseline/control for comparing performance of the dynamics model. If the dyn/?² is not significantly larger than this value, we cannot conclude to have obtained meaningful dynamics.
For the lower part of Table 1, we report the resulting metrics in Table 3, setting the number of forward steps n 10, and using either the original metric (Eq. 62), or the control (Eq. 63).
G.2 RESULTS
For the SLDS system, we can corroborate our results further: the baseline model obtains a dyn/?² of around 85% for single step prediction, both for the original and control metric. Our V-SLDS model and the ground truth dynamical model obtain over 99.9% well above the level of the control metric which remains at around 95%. The high value of the control metric due to the small change introduced by single time step, and should be considered when using and interpreting the metric. If more steps are performed, the performance of the V-SLDS model drops about 95.5% vs. chance level for the control metric, again highlighting the high performance of our model, but also the room for improvement, as the oracle model stays at above 99% as expected.
For the Lorenz system, we do not see a substantial difference between original mytri and dyn/t² control for any of the considered algorithms. Yet, as noted the main paper, -SLLDS the only dynamics model that gets high R.2 of 94.08%, vs. the lower 81 1..00% for a single LDS model, 40.99% for the baseline model. In other words, while DCL with the V-SLDS dynamics model falls short of identifying the true underlying dynamics for this non-linear chaotic system, without DCL wouldn't 't even identify the latents. We leave optimizing the parameterization the dynamics model identify non-linear chaotic systems for future work.
Table 3: Extended metries for dynamics models including additional variations on the dyn metrie where
Control is replacing f with the identity and 10 Steps is applying f (and f) 10 times, i.e., predicting 10 steps forward instead of only one step as done in the Original version.
data
model
% dynR? n=1 step Control
% dynR? 1=10 steps Original
f
P(E)
identifiable
Original
Control
SLDS
Normal
identity
85.47 +
84.54 + 7.31
2.78 + 9.34
-58622 + 7.22
SLDS
Normal
V-SLDS
99.93 + 0.01
95.15 + 0.68
95.53 + 0.47
-124.65 6.97
SLDS
Normal
GT
99.97 0.00
94.94 + 0.68
99.36 + 0.18
-129.32 6.95
Lorenz Normal
identity
X
27.02 4 8.72
27.17 +: 8.74
22,87 + 7.13
24.85 +: 7.14
Lorenz Normal
LDS
80.30 + 14.13
82.98 + 12.64
-13.07
+ 41.03
42.08 + 26.14
Lorenz
Normal
V-LLD
93.91 5.32
9
34.48 + 6.47
55.75 + 6.01
27

---

Published as a conference paper at ICLR 2025
H NON-INJECTIVE MIXING FUNCTIONS
Our identifiability guarantees (Theorem I, Def. 5) require an injective mixing function g(æ;) = yo On the first glance, this clashes with common requirements in system identification under partial observability. Specifically, let us consider system of the form:
E++1 = f(x)) + Et = Ax; t
(65)
y: = Ca; = (
where 3t RR" E RT C' E R with
m is non-square matrix that projects the states of the dynamical system into a lower dimensional space. Similarly, we may have I m with rCn < In those cases, C' and hence is non-invertible, and naively, the injectivity assumptions Def. 5 would fail to hold.
However, in practice this issue can be tackled through the use of a time-delay embedding. Specifically, we consider the following reformulation the system:
y
C
0
7+1
CA
V
V4-
CA2
Tt-r
Vi
(66)
Vt
CA
r-1
where vi
C Σ A' A'E
(67)
For sufficiently large time lag T, the linear map g (x) Oz = 5f will become injective again. This the case if O is full rank which holds if A is full rank, since bijective and therefore A is full rank square matrix. For example, if C' had rank m. then using at least r time steps would make ğ injective and our theoretical guarantees from Theorem would hold, up to the offset introduced by the noise v.
In practice, the change in latent space between different time steps might be small (especially when the time resolution of the system is very high). A practical way to avoid feeding increasingly large inputs, is to not feed in all time-lags 0... into the construction of O, but to subselect & time lags
Ti, Tke with Ti 0 and Tk т, and instead consider the system
y -
C
y +-2
CA
y/ =
Lt-7
+
V
(68)
Yt U
T
y Ut-Tn
CA
This system allows to have a sufficiently large context window (from t - i to f) to ensure injectivity, while keeping the input dimensionality of the model fixed. Note, when set we recover Eq. 66.
1 for each i,
Regarding the noise vector Hälvä al. (2021) recently showed that noisy and noise-free demixing problems can be mapped onto each other. While a full rigorous proof in conjunction with our Theorem is beyond the scope of this work, invoking Theorem of Hälvä al. (2021) to account for v is a promising avenue. Importantly, our empirical validation later on already shows that the model is practice indeed functioning even the presence of the noise distribution.
H.1 EXPERIMENTAL VALIDATION
We validate our theoretical considerations by two sets of experiments as closely related to the LDS setting from Table possible. We use types of mixing functions:
g(x) C, C2
(69)
g C2g' (Ciæ))
(70)
28

---

Published as a conference paper at ICLR 2025
b
linear
non-linear
0.8
0.8
points,
step window
Score
0.6
0.2
02
10
20
100
50
100
Time Delay (1)
Time De Melayi
Rank m-1
baseine DynCL
linear
m
non-linear
1.0
points, 100-step window
0.6
0.6
0.4
0.0
0.0
2
10
25
50
100
100
25
50
100
Time steps (K) flxed p-100
Time steps (K) for fixed t- 100
Figure 14: Non Non-injective mixing functions be successfully handled by time-lag embedding. in the first setting, we pass observations from consecutive time steps into feature encoder. empirical identifiability the latent space (R") for baseline (no dynamics) vs. DCL. (linear dynamics) as we increase n for a linear and e non-linear mixing function. d, to achieve injective mixing functions through time-lag embeddings, we here include the full 100-step length window, but only pass & equidistantly spaced points within this window of length empirical identifiability for baseline (no dynamics) DCL (linear dynamics) number of points in the context for a fixed r 100-step window and f for nonlinear mixing.
increase the
where C₁ F C2 E R are randomly sampled, g R" R" is a random injective and nonlinear function and m is the observable dimension, is the latent dimension and is the matrix rank. In line with the LDS experiments from Table 1, we set functions from Eq. 69 & 70 are then applied to the latents, including the parameter E (,) to restrict the rank and thereby dimensionality of the mixing functions to
and m 50. The mixing
We run experiments for different numbers of time steps that get passed to the encoder h. In the first setting, we use & consecutive time steps. this setting, we expect that more time steps than the theoretical minimum n/m are required, because the variation between consecutive time steps limited. To corroborate this hypothesis, we run a second variant where the length of the context window is fixed, and l equidistantly placed points selected to be fed in the encoder.
As in Table 1, we repeat each experiment times with different dataset and model seeds and report the 95% confidence interval.
H.2 RESULTS
Generally, we can see from Figure that we can successfully verify our theoretical considerations about weakening the injectivity constraint.
To begin with, we confirm that the default parametrization of DCL with only single time step cannot solve the demixing problem. Next, for the theoretical minimum of time steps (i = 6 for rank for rank 3) we can already observe a considerable improvement of the
1, = for rank 2, empirical identifiability for DCL in terms of the R22 Score: 16% 66% for rank 1, 30% - 78% for rank 2, 43% - 88% for rank (Fig. 14b). As mentioned above, the minimal amount of time steps required for the identifiability guarantees only hold for 2, which is not the in these experiments. As we increase the number of time points to 100, we can see that DCL approaches close perfect R. Scores: For linear mixing, we get the best results for i 100 with 99% R² for
29

---

Published as a conference paper at ICLR 2025
ranks 2, and 3. While for nonlinear mixing, averaged across seeds, get up to 86% for rank and 100, 97% for rank and 20, and 94% for = 20. In contrast, the baseline without dynamics model, does not benefit all from the additional time steps.
To further test our intuition about recovering injectivity, we include an experiment where the full 100-step length window is used, but only equidistantly spaced time steps are passed. In this setup, we much quicker recover acceptable identifability scores for the linear (Fig. 14e) and non- lon-linear mixing settings (Fig. 14f).
30

---

Published as a conference paper at ICLR 2025
I DYNAMICAL SYSTEMS WITH CONTROL SIGNAL
We have initially introduced the problem formulation of this paper (Equation as identifying the latent variables and the governing latent dynamics f from the observations yr for:
f(x,) Bu, E2 +++1 = ( + Vt-
(71)
with control signal its control actuator matrix B, system noise and observation noise
However, so far we have only explicitly considered autonomous latent dynamics (see Def. 5), which by definition did not include control input u4. In this section, show that the DCL framework works equally well in the presence of a control input and verify this empirically.
Being able to include control signal is crucial for many applications and common practice in systems identification literature. Without adding u, to the model, the task of identifying dynamics gets substantially harder the effects of the control signal become entangled with the intrinsic dynamics of the system. Additionally, only identifying the combined dynamics without factoring the effect of u, would make the framework less useful it would not allow predictions in the presence of new/different control inputs.
1.1 EMPIRICAL VERIFICATION
We extend our existing experiments for linear dynamical systems (LDS) by including control signal u, in the data generating process:
F++1 Ax, + Bu; + Ex
(72)
We train and evaluate four model variants:
Identity dynamics model, using the control input with with trainable B. The dynamics model is fit post-hoc,
Baseline:
DCL: with LDS dynamics model where B = O,
DCL w/ control: with LDS dynamics model and trainable B.
We choose the control signal u, be generated from:
Step function: composition of negative and positive step function, starting random time steps and random magnitudes.
Linear Dynamical System: u,+1 A, u, + Ex, similar to the LDS system used before for latent dynamics.
I.2 EXPERIMENT DETAILS
We generate three datasets with linear dynamics using a) no control, b) control following another LDS, and e) control following step function. Each dataset consists of 1000 trials, each trial is 1000 time steps long. The latent linear dynamics have intrinsic rotational dynamics with rotation angles 0; Uniform(0, 10) and control matrix B N(μ 0.01, Σ I -0001. The system noise e follows a standard normal with ag 0.001. The mixing function is the same nonlinear mixing function with 4 layers as for Table 1. We use 5-dimensional latents and have 50-dimensional observations. For the control following another LDS, we use the same sampling strategy for the parameters as for the latent dynamics. For the control following a step function, we pick two points Ti. T2 such that the T, Ti = 200 and that the step is centered within the trial including some random offset. We use different controls u, for every trial. We extend DCL with parametrization of the linear dynamics model that follows Eq. 72 and includes the control u, and trainable control matrix B. We use the same dynamics model for the baseline. However there, this only affects the post hoc dynamics fitting. We also train DCL with LDS model that does not include the control input. We train every model for 20k steps and besides that, we use the same hyperparameters as for training the LDS models in Table
31

---

Published as a conference paper at ICLR 2025
Table Empirical identifiability results when including both system noise and deterministic control signal The ground truth dynamics f are chosen as LDS, and the intrinsic dynamics model f is either an LDS, or the identity (baseline); Bu indicates whether the dynamics model includes the control (E) is Normal (small σ). Mean + std. are across 3 datasets and 3 experiment repeats.
Data
Model
Results
Control (u)
Bu
% %T2
&dyn/t" f
Step (ID)
identity
91.80 +: 1.15
87.27 1: 10.6
LDS
X
98.36 1 0.66
99.16 1 1.00
LDS
98.70 + 0.44
99.53 + 0.27
LDS (5D)
identity
74.17 + 13.8
76.47 + 7.51
LDS
X
98.26 + 0.17
99.87 + 0.04
LDS
98.10 = 0.35
99.85 + 0.08
Ground Truth
DynCL
Basel ne
fira)
0.0
FOOK
-10
1000
mum
0.0
-3.0
Figure 15: Visualization of dynamics inference in the presence of control signal u. LDS dynamics are complemented by ID step function signal, which projected SD using random matrix (not shown). The ground truth dynamics then show signatures move through latent space as we apply the step function. d, DCL uses a dynamics model and the control input u. e, the baseline uses only the control input u. The three rows show different dynamical systems, each plot is trial with 1000 steps.
autonomous dynamics when
(gray box). and
1.3 RESULTS
As shown in Table 4, when applying step function for the control signal (Step ID), DCL is able to identify both the latent space (98.7% R²) and the dynamics (99.5% dynR?). This result holds when the control signal is not used for training the model. However, baseline with identity dynamics is not able to identify the dynamical system with a substantially lower dyn 87.3% Nevertheless, the latent space can be estimated reasonably well, although not perfect (91.8 % R²), most likely because the availability of u converts the de-mixing problem into supervised learning problem (we have access to (u, y) pairs). This is reflected in the visualization in Figure 15: While the baseline (identity dynamics, but using u reasonably estimates the direction of u provided during training, the local dynamics cannot be fitted. We highlighted a gray box denoting the phase where = 0 to facilitate easier comparison.
To test DCL for more complex control signals, we also apply a full 5D LDS as the control signal u (Table 4, LDS 5D). Both latent space estimation and dynamics estimation performs par with the step setting (>98%), while the baseline fails to estimate either the latent space or the dynamics with R² values below 80%.
32

---

Published as a conference paper at ICLR 2025
J APPLICATION TO REAL- WORLD DATA
BCC Nures
LD9
aoooting
C Conssoonncy..o apranics
idnnity
Dessnsatarty
2.90
Equa Perfoma
26
0.50
0.60
0,70
0.80
0.50
1.00
Cons steney, idendit dynomics H9P
ports
daniy
Figure 16: Application to real-world neural data. (a) We leverage the Allen visual coding dataset using calcium imaging data (de Vries et al., 2020). We consider the subset of the data which is comprised of 10 movic repeats with 900 samples each 30Hz. We hold out the last trial for validation. Mouse icon from scidraw.io (Ann Kennedy), panel adopted from Schneider al. (2023). (b) We feed multiple time steps the model outlined in Appendix All experiments concat five equidistant points with maximum time lag at the sample locations (1, ..... L 20). (c) Quantitative comparison of DCL with identity vs. linear dynamics and qualitative overview of embeddings computed for the hyperparameters that (i) maximize consistency for LDS dynamics. (ii) have high consisteney for both models, (iii) maximize consistency for identity dynamics. Train embedding depicted as points (unsmoothed for LDS, smoothed for identity to show structure), test embedding as smoothed black trajectory. (d) Effect of smoothing applied to the embedding, left column depicts unsmoothed, right column smoothed embedding on train repeats. Upper panel is colored according to trial time, lower panel is colored according to overall time (train portion of data, test only shown above). In all embedding plots, the lest-embedding smoothed with 151 1asmmle filter is overlayed black line.
Here we evaluate DCL using a real-world dataset obtained from the Allen Institute (de Vries et al., 2020). The dataset contains recordings from awake, head-fixed mice as they viewed visual stimuli including three movies on a continuous loop. The recordings were collected using 2-photon calcium imaging. Our analysis focuses on paired data from 10 repetitions of "Natural Movie 1" comprised of 900 frames (Fig 16a). This exact dataset was also used by Schneider et al. (2023) and available as Allen-movee ceee-one-ceee caee-ca-VSSp 800- in the CEBRA software package.
J.1 METHODS
We train DCL with an LDS dynamics model. As our bascline, we use the identical training setup but with identity dynamics as in synthetic experiments. We use an MLP with three layers. The number of units for each layer scales with the embedding dimensionality d. The last hidden layer has 10d units and all previous layers have 30d units. We train on batches with 512 samples (reference and positive) and use 214 16384 negative samples. The model is optimized using the Adam optimizer (Kingma, 2014) with learning rate 10-4 We train DCL for 10k steps using the negative mean squared error as the similarity metric. For both models we also make use of the time-lag embeddings introduced in Appendix H (Figure 14), setting k = 5 and 7 20 (Figure 16b). Additionally, we vary the dimensionality d E {8, 16, 32, 64) of the embeddings. We also vary the time offset that determines the time step of the positive sample fpxx = het + i relative the time step of the reference sample and run experiments for E {1,5, 10, 15. 20, 25. 30}. We train each model 5 times with different initializations. For a shuffle control, we randomize the time dimension within each individual movie repeat (900 frames) and conduct the same training previously discussed. We train on the neural activity of the first repetitions (8100 samples, 270s) and use the 10th (900 samples, 30s) for evaluation. To evaluate the models, compute the consistency metric the R² metric between the embeddings - used by Schneider et al. (2023) between the five runs of the same model and training setup proxy for empirical identifiability. For the consistency calculation,
33

---

Published as a conference paper at ICLR 2025
a
b
1.0
shuffie
shuffre
no shufto
0.8
Consistency LDS dynamics
0.8
Cool Perfmmann
LDS
0.4-
0.4
shufti
0.2
hhuffla
shuffle
0.0
0.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Identity
dynamics
Consistancy, idantit dynamics (P9)
Consistancy, identity dynamics
Figure 17: Comparison of consistency with a shuffle control on (a) the train split and (b) the validation split. (c), example emheddings for hoth identity and LDS dynamics, in shuffle and no shuffle conditions.
fit linear regression model between all pairs of embeddings across training runs, and quantify the resulting x 4 comparisons using the R². We report the mean R² on the validation trial across all comparisons.
J.2 RESULTS
We compare the consistency scores between multiple runs of the same model training for DCL with LDS and our baseline (Figure 16c). Overall, DCL with a linear dynamics model has higher consistency scores compared training without a dynamics model for all settings except for time offset equal to 1, where results are on par. The positive effect of leveraging dynamics learning increases as we increase the time offset. Across all variations of the time offset, we additionally observe increase in consistency with increase in the dimensionality up to 32 dimensions, followed by a drop in consistency for 64-dimensional embeddings.
In addition to the quantitative performance improvements of DCL with LDS, we can also observe more structured embedding spaces for DCL with LDS Figures 16c and 16d compared to DCL without dynamics model. The embeddings of DCL with LDS exhibit clear manifold and follow relatively smooth trajectories, while the embeddings and trajectories DCL without a dynamics model are considerably more entangled. Considering the color code showing the relative time of each trial, we can also see that DCL with LDS recovers an embedding space in which each trial follows roughly the same circular motion as the other trials.
When removing temporal structure by shuffling (Fig. 17), neither embedding shows non-trivial structure and the consistency metric is low both the train (panel a) and validation set (panel b).
J.3 DISCUSSION
We observe the strongest improvements in consistency for those settings in which the time difference between the reference and positive sample is the largest (time offset 30). Under our assumed dynamics model, predicting further ahead in time results dynamics which differ increasingly from the identity dynamics. In our synthetic data experiments, we already observed a similar effect. The identity dynamics baseline was able estimate an embedding space when the noise dominated the system dynamics, but failed to estimate the embedding space as the dynamics became more prominent (Fig 5).
Note that our baseline model is similar but technically not identical - to the CEBRA-time models considered by Schneider et al. (2023) on the same dataset. Our results demonstrate potential improvements through the introduction of the dynamics model: First, we demonstrate that fitting embeddings Euclidean space using a negative mean squared error as the loss function, vs. the cosine similarity used by Schneider et al. (2023) benefits from the introduction of dynamics model. Second, the circular, repetitive structure movie repeats also emerges in this Euclidean space, most clearly in the presence of dynamics model (Fig, 16c, LDS). Third, we found it crucial to include the positive sample in the denominator which stabilizes training in the absence of normalized embeddings.
34

---

Published as a conference paper at ICLR 2025
K COMPUTATIONAL REQUIREMENTS
As stated in the main paper, we required 120 GPU days of compute for the experiments we ran for the paper. This does not include the initial period of prototyping and exploration that preceeded the final sweep of experiments. To provide more transparency and more detailed breakdown, we list the number experiments (= model trainings) run for each table and figure in the paper.
Result
Number of Experiments
Table 1
171
Figure 4 (SLDS)
162
Figure 5 & 7 (Lorenz)
675
Figure (Ablation)
648
Figure 8 (KDE)
1,125
Figure 9 (SLDS with vMF)
432
Total
3,213
Table 5: Number of experiments per table/tigure.
For 120 GPU days this comes out at an average 53 minutes per experiment that made it into the final paper. However, the actual runtime of an average DCL training is 15-20 minutes for settings equivalent to the experiments in Table 1. This difference to what we report as the overall average compute time per experiment can be attributed several factors: (1) approximately one-third of experiments involved KDE estimation which required 4-6x longer training times, (2) extensive evaluation and metric computation for debugging and reporting purposes added additional overhead, and (3) additional experimental iterations that did not make it into the final paper but contributed to the total compute time of the final sweep.
35

---

Published as a conference paper at ICLR 2025
L ON COMPONENT-WISE VS. LINEAR IDENTIFIBBILTTY
In the context of non-linear ICA, the mean correlation coefficient (MCC) is frequently reported as a measure of component-wise identifiability. In non-linear ICA, it is typically assumed that set of independent sources $1 (t)....... (n) is passed through mixing function to arrived at the observable signal (cf. Hyvarinen & Morioka, 2017). contrast, in our work the sources are not independent, but are conditioned on the previous time step and the passed through dynamics model. This is conceptually similar to the conditional independent assumption in Hyvarinen et al. (2019) with auxiliary variable f(z), but with the distinction that at training time, we do not have u available, only æ which requires the use of a dynamics model.
Because of these distinctions in the generation of the latent space, can generally not expect component-wise identifiability (Theorem 1) but will instead obtain linear identifiability. Related work in non-linear ICA likewise can only provide linear identifiability for comparable Gaussian case (Hyvarinen et al., 2019; Zimmermann et al., 2021; Schneider et al., 2023). However, if we assume access to the dynamics model (but the actual latent space), is possible to reduce ambiguity in the latent space, and assuring component wise identifiability.
In Table 6 we compare MCC and R2² across datasets (SLDS and Lorenz) to extend Table of the main paper. As expected, we observe non-perfect MCC score for all models except for the SLDS model which is provided with the ground-truth dynamics. Due Eq 29 this strenghten the guarantee to component-wisg wise identifiabilily, yielding an MCC of close to 100%.
Data f
Model f
MCC
&7?
SLDS
identity
0.59 = 0.06
76.80
7.40
SLDS
SLDS
0.70 = 0.05
99.52 0.05
SLDS
GT
1.00 = 0.00
99.20 + 0.10
Lorenz
identity
0.34 = 0.07
41.00 8.57
Lorenz
LDS
0.68 = 0.14
81.20 + 16.9
Lorenz
SLDDS
0.78 = 0.13
94.08 + 2.75
Table 6: SLDS and Lorenz dataset from Table with the addition of the MCC metric.
36

---

Published as a conference paper at ICLR 2025
M COMPARISON TO ADDITIONAL TIME-SERIES MODELS
The baseline- DCL without a dynamics model employed in all experiments in the main paper is technically& CEBRA-time (Schneider et al., 2023) model which was originally designed for time- series inference. Note that we use the negative mean squared error as the similarity measure in almost all experiments, and include general improvements for estimation Euclidean embeddings also in the baseline model (positive sample in the denominator of the InfoNCE Ioss, additional negative examples). Other modes of CEBRA-time (using a cosine similarity, etc.) were not prominently considered in this work. Other popular contrastive learning methods include tme-contrastive learning (TCL; Hyvarinen & Morioka, 2016), permutation contrastive learning (PCL; Hyvarinen & Morioka, 2017). More recently, VAE models designed for time-series analysis and dynamies learning were proposed, such as Temporally Disentangled Representation Learning (TDRL; Yao et al., 2022).
Naturally, these methods propose different data generating processes, and the empirical performance of DCL as well as these comparison methods will strongly depend on whether these assumptions met. For instance, TCL requires non-stationary independent sources, PCL requires stationary independent sources, and TDRL uses non- con-aaamettric transition models with non-Gaussen noise. Hence, in general, it is not possible to fairly compare all these methods as they have different regimes of operation. We still include comparison of these methods and DCL. for dynamics inference.
M.I VERIFYING BASELINES
Yao al. (2022) published benchmarking setup for these algorithms'. We adapted the TDRL codebase minimally and provide reference implementation for our experiments in our official code release. Our implementation can be diffed against the original TDRL code highlight the minimal code changes performed for rumning the benchmarking suite. We leverage this codebase to run verified baseline algorithms on our SLDS dataset. First, we ensure that we can reproduce the results from Yao et al. (2022). We report the results in Table 7 for the "changing" experimental setting.
We perform runs with different seeds for the exact hyperparameter configurations reported in the TDRL codebase. Remaining differences in the in numbers might be attributed to discrepancies between the paper and the code distribution or the choice of different random seeds, as we observed large variances in some of the models. We label these models with "-B" to indicate the base configuration provided by the public code base. We report the full results Table 7.
MCC
Model
Reproduced
Reported
PCL.-B
0.535 + 0.030
0.599 + 0.041
TCL-B
0.367 + 0.018
0.399 + 0.021
TDRL-B
0.910 1 0.067
0.958 1 0.017
Table 7: Verification on "changing" experiment Setting from Yao al. (2022)
M.2 EXPERIMENT DETAILS
Both TCL and TDRL make use of categorical context variable indicating changes in the distributions of the true latents. To make the comparison to our framework as fair as possible, we choose the SLDS datasets allow for similar context variable in form of the mode/state sequence that modulates the switching between linear dynamics, Our dataset is comprised of 5 modes, which corresponds to the same number of categories the models from Table already use.
Base models. To be able to apply the baseline models to our SLDS setting, we only change their base configuration in two required ways: We increase the input dimension from 50 to match the observation produced by the SLDS and reduce the latent dimension from 8 to 6.
Large models. Because PCI. is the closest match to our existing baseline (CEBRA-time) and our encoder architecture is equal to the baseline architecture, we introduce an additional variant "PCL.-I. (L=Large) to match the number of parameters as close as possible. We do so by increasing the hidden dimension of the PCL encoder model from 50 to 160 and reduce the number of layers from 4 to 3,
*Code: https: /tt::://itttttt com. www.o///ww///////..oaa// (MIT License)
37

---

Published as a conference paper at ICLR 2025
low noise
high noise
Model
MCC (%)
R² (%)
MCC (%)
R² (%)
TCL-B
36.07 2.27
66.56 + 3.87
37.21 3.66
61.75 + 12.56
PCL-B
68.28 + 2.40
91.33 0.85
66.86 3.16
77.99 = 3.93
PCL-L
68.02 + 2.77
00.9211.1
69.67 3.86
80.88 = 1.38
TDRL-B
64.34 6.01
83.85
62.93 5.37
80.90 7.82
TDRL--
63.51 4.87
84.01 7.98
62.65 + 6.29
81.40 = 6.42
CEBRA-time
59.46 5.84
76,80 7.40
65.71 1 4.99
98. 68.66 = 0.19
DCLASLDS
69.62 4.78
99.52 + 0.05
68.76 + 5.05
98.95 = 0.08
DCL+GT SLDS
99.51
0.06
99.20 + 0.10
98.57 + 0.16
97.82 = 0.17
Table 8: Baseline results for TCL, PCL, and TDRL models on switching linear dynamics datasets. The low noise setting is equivalent to Table 1. For the high noise setting (low At), we use larger noise and lower rotation angles, setting de 0.001, max( 5.
effectively increasing the number of parameters by factor 5. Because TDRL can be considered the most promising baseline candidate (beside CEBRA-time) based on the results from table 7, also double its encoder size from using hidden dimension 128 to 256, resulting in the "TDRL-L" baseline model.
Dataset. Leverage two versions of the SLDS dataset used in the main paper. First, we apply it to the exact setting of the SLDS in our Table to compare against our default setting with dynamics noise oe = 0.0001 and rotation angles max (0:) = 10. Additionally, since our main baseline (CEBRA-time) performed best on datasets with lower Á where the noise dominates over the dynamics, we also compare against an SLDS dataset generated with larger dynamics noise de 0.001 and smaller rotation angles max (0.) = 5 (see Figure 4b). We generate different versions of each dataset using different random seeds and on each dataset we train 3 models with different seeds, resulting in models for each baseline and for each of the two settings. We train every baseline model for 50 epochs until the training time reaches 8 hours.
Metrics. We compute the Mean Correlation Coefficient (MCC) as well as the R metric used for Table 1 during training. For the baselines run with the public code base from Yao et al. (2022), follow their reporting strategy and report the best MCC complemented by the the best R² achieved at any point during training to make the baseline appear even stronger. For DCL. models and our default baseline, we report the MCC and R² based on the last model checkpoint as in the main paper.
M.3 RESULTS
We outline results for all benchmarked algorithms in Table 8.
In the low noise setting, DCL with the SLDS dynamics model achieves an R² of 99.5%. The next best baseline algorithm is the PCL base model, with a maximum Rd of 91.3%. While both DCL and PCL are learning by contrasting samples across time in the time series, only DCL with the SLDS dynamics model can fully model the ground truth dynamical process. In contrast, the score function in PCL is setup model variations along independenl latent dimensions.
Interestingly, PCL outperforms CEBRA-time, which is equivalent to running DCL. without a dynam- ies model (76.8%). This indicates that the score method PCL (component-wiss linear transfor- mations) outperforms the score method in CEBRA-time on this dataset (negative squared Euclidean distance). would be interesting to combine the scoring method in PCL with the dynamics model in DCL for further improvements on non-lineat dynamics settings.
In the high noise setting, DCL with SLDS dynamics achieves comparable performance as CEBRA- time, as outlined in the main paper (99.0% vs. 98.7%). In this setting, PCL performs worse, potentially because the score function in CEBRA-time is better matched to the dominating Gaussian system noise.
In all cases, MCC is not a meaningful metric, with highest scores ranging around 60-70%. exception is training DCL with the underlying ground truth dynamics system, which achieves component-wise identifiability and an MCC of 99.51%.
38


Reviews:
[{"review_id": "a4ZWw6SA5l", "reviewer": "Reviewer_KF3j", "summary": "The main contribution of this paper is to provide identifiability theory for contrastive learning on time-series data with non-linear mixing, in the same spirit as time-contrastive learning (Hyvärinen & Morioka, 2016) for non-linear ICA. However, the authors discard the independence assumptions typically made in non-linear ICA with respect to the latent variables, and instead define a dynamical system as the data generating process. The proof operates under the assumption that the mapping from latent states to observables is injective but not necessarily linear, which is exploited to show that the composition of mixing and de-mixing by the model is an affine transform. As such, the estimated dynamics via contrastive estimation identify the true dynamics up to affine transformation in the latent space. There are experiments that corroborate the validity of this approach.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Applying contrastive learning to recover latent dynamics is itself a relatively new approach and the paper is well organised. The proofs use standard jacobian analysis tools and is easy to follow.", "weaknesses": "The paper needs refinement, with minor typos and inadequately captioned figures. While studying the identifiability of time-series contrastive learning might be novel, all the technical tools require carefully controlled assumptions and specific behavior of Jacobians under contrastive loss minimization, which typically do not hold in practice. Nonetheless, such assumptions are common in the literature on the identifiability of dynamical systems from observed time series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OA3ukAhU3z", "reviewer": "Reviewer_GCbH", "summary": "The paper explores the use of self-supervised contrastive learning for dynamic system identification. It connects self-supervised learning (SSL) with identifiable representation learning, showing that SSL can identify system dynamics in latent space. The authors propose a model to uncover linear, switching linear, and non-linear dynamics under a non-linear observation model, providing theoretical guarantees and empirical validation.", "questions": "See the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The connection between contrastive learning and dynamic system identification is novel and could lead to simple encoder-only implementations favored in practice.\n2. The synthetic experiments, especially the ablation studies, are extensive and investigate many aspects of the theoretical results.", "weaknesses": "1. The contributions (comparison with previous work), theoretical techniques, and novelty are very lightly discussed. I would appreciate a detailed discussion comparing this work's conditions with those in recent literature on temporal causal representation learning (e.g., [1] and its follow-up work). This would aid the contextualization of this work and make the contribution more transparent. \n\n2. This work strikes me as theoretically oriented. There is a lack of discussion on the theoretical conditions, limitations, and implications. Just to name a few: \n\n(a) the theorem requires the length for each time series to be infinite, which appears to be quite a strong assumption and not necessitated in recent work (e.g., [1], admittedly I am not an expert and would appreciate correction if I have misunderstood the statements). Why is this necessary, and would it be possible to show results for finite-length series?\n\n(b) How does the control distribution influence the identification results? what conditions should it meet (does it have to be a time-independent Gaussian distribution)?\n\n(c) Line 175: what is the significance of $A$ -- is it previously defined? Could you comment on lines 175-177?\n\n3. Some minor issues:\n\n(a) Line 145: what is (...)?\n\n(b) Line 165: shouldn't $L^{\\top} L$ be a matrix?\n\n\n[1] Temporally Disentangled Representation Learning. Yao et al. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ap4NehPA7i", "reviewer": "Reviewer_4X1L", "summary": "In this paper, the authors propose a system-identification scheme for non-linear observations of non-linear time series data. In particular, they propose a modified contrastive learning set-up that posits linear latent dynamics. Compared to prior works in (time) contrastive learning, this directly enforces a notion of sequential temporal consistency, and seems to provide some benefit in system identification settings. Some supporting theory is provided, demonstrating that if the underlying dynamics are linear and invertible, then the proposed method asymptotically recovers the true dynamics up to affine ambiguity. For general non-linear systems, a (soft) switched-linear system heuristic is proposed, where Jacobian linearizations are applied at user-provided reference points.", "questions": "My main questions can be summarized as follows:\n\n1. What is the marginal utility of this method rather than various other latent nonlinear dynamics estimation methods, e.g. (Watter et al., 2015) (which in fact also imposes locally linear latent dynamics), which do not make strong assumptions on identifiability?\n\n2. How does enforcing the identifiability/invertibility conditions in this paper affect the method's performance in partially observed settings? This could be as simple as the LQG setting detailed above. Does this strong inductive bias translate to large errors when it is not satisfied (which is typically the case when only provided with observations of a ground-truth belief/latent state)?\n\n3. As detailed above, what is the role of the control input? What is the effective difference of the proposed setting and an autonomous (latent) dynamical system $x_{t+1} = f(x_t) + \\epsilon_t$?\n\n4. From a practical perspective, how are the reference points for computing first-order linear approximation in the switching case chosen? Also, do these need to be recomputed per iteration, since the parameterization of $\\hat{\\mathbf{f}}$ changes per iteration? \n\n**References:**\n\nWatter et al., \"Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images\", 2015.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Automatic identification of latent variables or dynamics is of critical importance in modern machine / reinforcement learning. The method the authors propose follows a line of self-supervised methods in contrastive learning. In comparison to its closest relative in time-contrastive learning (Hyvarinen and Morioka, 2016), the proposed method is seemingly more well-fit for fitting non-linear time-series data by fitting a latent time-series, rather than predicting a categorical label as in the aforementioned paper.\n\nSince a main inductive bias built into the base method is that the latent dynamics are linear, the proposed method of iterative Jacobian linearizations is a sensible adaptation, and seems to benefit performance significantly.\n\nNumerically, the proposed method appears to make contrastive methods more robustly performant.", "weaknesses": "In my opinion, the paper leaves quite a few critical questions unanswered, and in general suffers from a lack of polish. In its current state, I cannot recommend the paper for acceptance. The main weaknesses in my eyes are the following:\n\nThe paper claims to perform latent nonlinear system identification. This is a key desideratum in various fields such as reinforcement learning and continuous control, and thus has a rich history and literature. However, the assumptions in this paper--and notably how these inductive biases propagate to the algorithm design--severely restrict the applicability of the method without further evidence. Notably, a design assumption in this paper is that the observer function (i.e.\"mixing function\") is invertible. This is a very strong assumption in the context of *non-linear system identification*, where even the foundational theory of linear system identification does not presume: in the Linear-Quadratic Gaussian (LQG) model, where the underlying state evolves linearly $x_{t+1} = Ax_t + Bu_t + w_t$, and observations are a linear function of state $y_t = Cx_t + v_t$ (ignoring the control input term for simplicity), the classical set-up has $d_y < d_x$, such that the observations are per-timestep a low-dimensional measurement of the underlying state. This immediately rules out the mixing function $g(x) = Cx$ being invertible, and this is precisely the motivation for notions such as observability/detectability. Partial observability presents the key challenge in non-linear sysID or reinforcement learning. In particular, it is well-known in controls and RL that ignoring partial observability and imposing a Markovian model (which this paper does implicitly by enforcing the state estimate as a function solely of the current observation) can lead to very undesirable outcomes. In the contrastive learning literature, partial observability is usually not a central issue, often because it is irrelevant for the motivating application (e.g. in computer vision), but one must address this problem for time-series data. In fact, the cited Time-Contrastive Learning method (Hyvarinen and Tomioka, 2016), despite making the same assumption in theory, actually propose a method that is more amenable to partial observability, since they predict categorical labels to *chunks* of observed data.\n\nRegarding the polish of the paper, there are various typos and lacking definitions that make the paper hard to parse at times. The minor ones that I have caught are listed below. A particularly confusing point is the role of the control input $u_t$. The paper presents the control input as entering the latent dynamics directly. However, it is typically the case that the control input enters the state through a (possibly state-dependent) actuation matrix $\\mathbf B(x_t) u_t$. In any case, how the control input enters the dynamics should be dependent on the parameterization of the dynamics, e.g. the affine ambiguity in $\\mathbf L$ in the paper, which is not reflected in the authors' method as far as I can tell. Furthermore, it is unclear if the control input is available to the learner (which is usually the case in sysID), or if it is playing the role of stochastic noise, which eq (9) seems to suggest compared to eq (1). In either case, what role is the control input playing here: in the authors' set-up, there is no need to learn the actuation matrix, and the experiments involve learning a low-noise, nearly deterministic Lorenz system, which rules out some persistency of excitation effect (Tsiamis and Pappas, 2019).\n\n**Minor comments/typos:**\n\nFigure 1: x -> $x$\n\nPage 3: \"linear identifiability (...)\", missing eqref?\n\nTheorem 1: \"bijective dynamics model $\\mathbf f$\", should probably mathematically define what that means.\n\nTheorem 1: $\\lambda$ is not defined in the main paper, only in the appendix.\n\nCorollary 1: \"$\\hat{\\mathbf f} :=1$\", seems to be bad notation.\n\nBeginning of Sec 4: \"non-lineary\" -> \"non-linearly\"\n\nEquation (7): where is $g_k$ defined? Possible hash collision with mixing function notation.\n\nTable 1: should probably introduce acronym \"LDS\" = Linear Dynamical System somewhere\n\nTable 1: What does LDS$\\downarrow$ mean?\n\nTable 1: What do $\\mathbf L$, $\\mathbf L'$, $(\\checkmark)$, $\\mathbf I$ in the theory column indicate?\n\nBetween (11) and (12): \"tailor\" -> \"Taylor\"\n\nBefore Sec 5: \"matices\" -> \"matrices\"\n\nImplementation paragraph: possibly missing number of A100 cards?\n\nEq (23): where is $p_u$ defined?\n\nEq (25): what does $p_{D}(y)$ denote precisely?\n\nAfter Eq (50): \"which is probably still fine because $\\exp(-\\|\\mathbf L \\cdot \\|^2)$ is a valid kernel function (?)\". This probably needs to be formalized/reworded.\n\n**References:**\n\nAnastasios Tsiamis and George J. Pappas, \"Finite Sample Analysis of Stochastic System Identification\", 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6jWRbul9r", "reviewer": "Reviewer_3gxR", "summary": "The paper \"Self-Supervised Contrastive Learning Performs Non-Linear System Identification\" explores contrastive learning (CL) for identifying non-linear temporal representations. It presents proof of the identifiability of latent variables up to a linear transformation, removing the requirement for independent noise. The proposed model, DynCL, is validated using synthetic data to support the theoretical findings. Additionally, the authors introduce a model called delta-SLDS, designed to capture switching between linear and non-linear dynamic systems.", "questions": "- It is confusing why it requires 120 GPU days on A100 for these synthetic data. Methods like TCL require only dozens of minutes for one synthetic data. What is the detailed model structure with approximate parameter size?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The theorem is both interesting and novel, demonstrating that the learned latent variables can be identifiable up to linear transformations even in the absence of independent noise, provided that some other assumptions are met.\n- This theorem provides valuable insights into the mechanisms underlying contrastive self-supervised learning methods.\n- A lot of ablation studies and visualization experiments are conducted, which makes the paper more convincing.", "weaknesses": "- Notations are not clear. For example, in Eq (3), the meanings of $x, x', x''$ should be mentioned in advance.\n- In theorem (1) and its proof, the assumption is not aligned with Equation (1-2), where all noise disappears. Further discussion is required.\n- The difference in theorem part should be compared with previous works on CL more detailed, since it is a work focusing on theorem. Making the difference more clear will make it more readable.\n- In experiment parts\n    - Baselines like TCL should be compared\n    - By identifiability, some metrics like MCC should be compared even though they are not component-wise identifiable.\n- Lots of typos:\n    - line 133 (supp figure), line 145 (...), seems unfinished part\n    - line 250: tailor -> talyor\n    - line 637: Theorem 2 -> Theorem A1\n    - footnote of page 13: broken reference  \n    - line 659, 665: missing reference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ujmKw351K", "reviewer": "Reviewer_qeKY", "summary": "This paper focuses on contrastive learning (CL) methods for dynamical systems.\nThe authors show that under certain assumptions CL performs system identification and can therefore uncover the latent dynamics of the data.\nThe theoretical findings are applied to switching linear dynamics and non-linear dynamics, and are demonstrated from an empirical point of view using simulated data.", "questions": "1. You use the Gumbel-softmax to approximate the argmin. In my experience, the proper turning of the schedule of its temperature parameter is quite challenging. Is it something you noticed as well?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall this is an interesting paper, that gives an insight on why CL techniques are effective for system identification\n2. The introduced CL variant is well presented and theoretically grounded, and could inspire further theoretical research and models\n3. DynCL can effectively identify latent states and system dynamics in the experiments on simulated data\n4. The authors present a good selection of ablation studies to demonstrate the impact of the different modeling/parametric choices", "weaknesses": "MORE REALISTIC EXPERIMENTS\n\nAs stated by the authors in the limitations of this work, the focus of this paper is only on simulated data. While I understand their point of view, and I also agree that the theoretical contribution/simulated experiments are also valuable by themselves, I fear that the impact of this work in this current state will be more limited than it could be if there was a better demonstration of real world applicability.\n\nYou could for example try to apply your method to some of the datasets used in \"Discovering State Variables Hidden in Experimental Data\" (https://www.cs.columbia.edu/~bchen/neural-state-variables/).\n\nIf the above is too challenging to achieve, you should at least try to discuss more in detail what each of you theoretical assumptions means in practice, and what you expect to happen if they are not met in real-world experiments. For example, the fact that $p(u_t)$ is a normal distribution seems quite strict in many applications. \n\nBASELINE\n\nThe baseline you use in your experiment seems quite weak, as it does not even use a dynamics model. Have you tried other approaches, for example models doing next-token prediction tasks?\n\n\n\n\nCLARITY\n\nThere are several missing definitions/clarifications in the paper that make it a bit harder to follow:\n1. N in (3) is not defined\n2. \"Supp figure\" in line 133 is unclear\n3. Not sure what \"(…)\" in line 145 means\n4. The name $\\nabla$-SLDS is never formally defined\n5. In table 1 you have a column called \"theory\" with different options. What do these option represent exactly?\n6. The abbreviation \"GT\", which I assume stands for ground truth is used in many places but never defined\n7. What is $\\pi$ in equation (8)?\n8. The vMF abbreviation is never defined\n9. The DynCL results from Table 1 should be discussed more in depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)