On the Identification of Temporal Causal Representation with Instantaneous Dependence
Authors: Zijian Li
Yifan Shen
Kaitao Zheng
Ruichu Cai
Xiangchen Song
Mingming Gong
Guangyi Chen
Kun Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: Causal Representation Learning
Instantaneous Dependency
Identification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Temporally causal representation learning aims to identify the latent causal process from time series observations, but most methods require the assumption that the latent causal processes do not have instantaneous relations. Although some recent methods achieve identifiability in the instantaneous causality case, they require either interventions on the latent variables or grouping of the observations, which are in general difficult to obtain in real-world scenarios. To fill this gap, we propose an \textbf{ID}entification framework for instantane\textbf{O}us \textbf{L}atent dynamics (\textbf{IDOL}) by imposing a sparse influence constraint that the latent causal processes have sparse time-delayed and instantaneous relations. Specifically, we establish identifiability results of the latent causal process based on sufficient variability and the sparse influence constraint by employing contextual information of time series data. Based on these theories, we incorporate a temporally variational inference architecture to estimate the latent variables and a gradient-based sparsity regularization to identify the latent causal process. Experimental results on simulation datasets illustrate that our method can identify the latent causal process. Furthermore, evaluations on multiple human motion forecasting benchmarks with instantaneous dependencies indicate the effectiveness of our method in real-world settings.

Published as a conference paper at ICLR 2025
ON THE IDENTIFICATION OF TEMPORALLY CAUSAL REPRESENTATION WITH INSTANTANEOUS DEPEN- DENCE
Zijian Lif** Yifan Shen"
Kaitao Zheng Ruichu Cai Xiangehen Song' Mingming Gong"
Guangyi Chen Kun Zhangi*
iCanegie Mellon University, Pittsburgh PA, USA
*Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
!Guangoogg University of Technology. Guangzhou, China
"The University of Melbourne
ABSTRACT
Temporally causal representation learning aims to identify the latent causal process from time series observations, but most methods require the assumption that the latent causal processes do not have instantaneous relations. Although some recent methods achieve identifiability in the instantaneous causality case, they require either interventions the latent variables or grouping of the observations, which are in general difficult to obtain in real-workd scenarios. To fill this gap. we pro- pose an IDetification framework for instantaneOus Latent dynamics (IDOL) by imposing sparse influence constraint that the latent causal processes have sparse delaye and instantaneous relations. Specifically, we establish identifiability results of the latent causal process up to Markov equivalence class based on sufficient variability and the sparse influence constraint by employing contextual information. We further explore under what conditions the identification can be extended to the causal graph. Based on these theoretical results, we incorporate a temporally variational inference architecture estimate the latent variables and a gradient-based sparsity regularization to identify the latent causal process. Ex- perimental results on simulation datasets illustrate that method identify the latent causal process. Furthermore, evaluations on human motion forecast- ing benchmarks indicate the effectiveness in real-world settings. Source code is available at htnps//gghibbbbcm//DDDMiI/BBB///
INTRODUCTION
Time series analysis (Zhang et al., 2023 Tang Matteson 2021 Li al. 2023a Wu al. 2022
Luo & Wang 2024), which has been found widespread applications across diverse fields such as weather (Bi et al. 2023 Wu et al. 2023b), finance (Tsay 2005 Huynh et al., 2022), and human capture the underlying patterns (Wang et al. 2019 Jin et al. (022) behind the time series data. To achieve this, one solution is to estimate the latent causal processes (Tank et al. 2021 et al. 2023b Zheng & Kleinberg 2019).
activity recognition Yang et al. 2015 Kong & Ful 2022).
However, without further assumptions (Locatello et al. 2020). it is challenging to identity latent causal processes, i.e., ensure that the estimated latent causal process is correct.
Researchers have exploited Independent Component Analysis (ICA) (Hyvärinen & Oja 2000 Comon 1994 Hyvärinen 2013 Zhang & Chan 2007), where observations are generated from latent variables via a linear mixing function, to identify the latent causal process. To deal with nonlinear cases, different types of assumptions including sufficient changes (Yao et al. [02T Khemakhem al.
2020a| Xic et al.) 2022b) and structural sparsity (Lachapelle et al. [022 Zheng et al. (022) are proposed meet the independent change of latent variables. Specifically, several works leverage auxiliary variables to achieve strong identifiable results of latent variables (H (H"alv"s a & Hyvarinen
2020 Hälvä et al. 2021 Hyvarinen & Morioka 2016 2017 Khemakhem et al. 2020a Hyvarinen
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
Latent/Observed Variables
Mixing Process
Latent Causal Process
f-11
ZC1
L1
ZE,1
7
2
Z
xt-11
Xx,1
xt-12
X1,2
Xx
Xx,2
1-222
Xx2
(b) G- B-CaRL with Grouping Observations
(c) IDOL with Sparse Latent Process
(a) iCITRIS with Interventions
Figure 1: Three different data generation processes with time-delayed and instantaneous dependencies. (a) iCITRIS (Lippe et al. 2023) requires intervention variables i, for latent variables (the gray nodes with blue rim.). (b) G-CaRL (Morioka & Hyvärinen| 2023) assumes that observed variables can be grouped according to which latent variables they are connected to (the blue dotted lines), (c) IDOL requires sparse latent causal process (the blue solid lines).
al. 2019). To seek identifiability an unsupervised manner, other researchers (Ng et al., 2024, Zheng & Zhang 2024) propose the structural sparsity on the generation process from latent sources to observation. However, these methods usually do not have instantaneous relations, e.g., the latent variables mutuaaly independent conditional their time-delayed parents at previous timestamps. Hence, further assumptions are used for identifiability. For example, Lippe et al. (2023) demonstrate identifiability by assuming that there exist interventions latent variables, and Morioka & Hyvärinen (2023) yield identifiability with the assumption of the grouping of observational variables. Recently, /Zhang al. (2024) use the sparse structures of latent variables to achieve identifiability of static data under distribution shift. Beyond it, in this work, we demonstrate that temporal identifiability benefits from the sparsity constraint the causal influences in the latent causal processes by employing the temporal contextual information, i.e., the variables across multiple timestamps. Please refer to Appendix A and Diro further discussion of related works, including the identification of latent variables and instantaneous time series.
Although these methods (Lippe al., 2023 Morioka & Hyvärinen, 2023) have demonstrated
the identifiability of latent variables with instantaneous dependencies, they often impose strict assumptions on the latent variables and observations. Specifically, as shown in Figure [ (a), iCITRIS (Lippe et al. 2023) reaches identifiability by assuming that interventions L, i.e. the gray nodes with blue rim, on the latent variables, which is expensive and difficult in practice. For instance, considering the joints as the latent variables of a human motion process, it is almost impossible to intervene in the joints of the knee when a human is walking. Moreover, as shown in Figure (b), (b), G-CaRL (Morioka & Hyvärinen 2023) employs the grouping of observations identify the latent variables, i.e., the blue dotted lines. However, the grouping of observations is usually expensive, for example, grouping the sensors on a human body from the joints of skeletons requires massive of expert knowledge and labor. Therefore, it is urgent to find a more general approach with an appropriate assumption to identify the latent causal process with instantancous dependencies.
To identify the latent variables in more relaxed assumptions, we present a general IDentifiation framework for instantaneOus Latent dynamics (IDOL) with the only assumption of the sparse causal influences in the latent processed named "sparse latent process" as shown in Figure (c). where the transition process denoted by the blue solid lines sparse. It is worth noting that the sparse latent process assumption is common and naturally appears in many practical time series data. In the example of human motion forecasting, a skeleton of the human body, which can be considered to be a latent causal process, is usually sparse. This is because there are a few connections among the joints of the human body. Based on the latent process sparsity, we first establish the relationship between the ground truth and the estimated latent variables by employing sufficient variability. Sequentially, we propose the sparse latent process assumption to achieve the component- wise identifiability of latent variables by leveraging the contextual information of time series data, which implies the
2

---

Published as a conference paper at ICLR 2025
identification of Markov equivalence class. Furthermore, causal graph when the endpoints of instantaneous edges do not share identical time-delayed parents, ie, the example in Figure I (c) preciscly satisfies this condition. Building upon this identifiability framework, we develop the IDOL model, which involves a variational inference neural architecture to reconstruct the latent variables and gradient-based sparsity penalty to encourage the sparse causal influences in the latent process for identifiability. Evaluation results on the simulation datasets support the theoretical claims and experiment results on several benchmark datasets for time series forecasting highlight the effectiveness of our method.
extend to the identification of the
PROBLEM SETUP
2.1 TIME SERIES GENERATIVE MODEL UNDER INSTANTANEOUS LATENT CAUSAL PROCESS
We begin with the data generation process shown in Figure 1 (c) where the latent causal process is sparse. To facilitate clarity, we adopt the terminology widely used in ICA literature like observation X,, latent variables 2, and mixing function g (Hyvarinen & Morioka, 2016). Suppose that have time series data with discrete timestamps X [XI."*, Xt,".. XT y. where x, is generated from latent variables 2, 2 C R via invertible and nonlinear mixing function g. We have:
=g(zz)..
(I)
Specifically, the i-th dimension the latent variable žτ,i is generated via the latent causal process, which is assumed to be related to the dime-delayed parent variables Paa(2t,) and the instantaneous parents Pag (2).s). Formally, it can be written via a structural equation model (Neuberg 2003) as
2 (P (2r,,,, es,)) with 6,,i ~ Pea:
where é,, denotes the temporally and spatially independent noise extracted from a distribution Pe. To better understand this data generation process, we take an example of human motion forecasting. where the joint positions can be considered as latent variables. In this case, the rigid body effects among joints result in instantaneous effects, and the human movement trajectory recorded by the sensor observed variables. As result, we can consider the process from latent joint positions to movement trajectory as the mixing process.
2.1.1 IDENTIFIABILITY OF LATENT CAUSAL PROCESSES
Based the aforementioned generation process, further provide the definition of the identifiability of latent causal process with instantaneous dependency in Definition Moreover, if the estimated latent processes can be identified at least up to permutation and component invertible transfor- mation, the latent causal relations are also immediately identifiable up to a Markov equivalence class (Spirtes al. 200T). We further show how to go beyond the Markov equivalence class and identify the instantaneous causal relations with mild assumption in Theorem
Definition 1 (Identifiable Latent Causal Process (Yao al. 2022 2021 Let (x1..... .,T...,,,))))) be
sequence of observed variables generated by the true latent causal processes specified by (fi,p(e;).e given in Equation [ and ( learned generative model (fr, ((c;). g) is observational equivalent to matches the data distribution pf.p..g ((x)Tm)
(frs p(c)) g) if the model distribution Pj.... ({x)T=)) ((x)
for any value of {x]T= We say latent causal processes identifiable observational equivalence can lead to a version of the generative process up to permutation n and component-wise invertible transformation T:
Pis.e., & ((x)T=1) PfePe ((x)
g=goroT.
Once the mixing process gets identified, the latent variables will be immediately identified up to permutation and component-win wise invertible transformation:
2. g-- (x:) (T-)
(x:) (T- o x====")) (g (xx)) (T o ())))) (2:).
(4)
3 IDENTIFICATION RESULTS FOR LATENT CAUSAL PROCESS
Given the definition of identification of latent causal process, we show how to identify the latent causal process in Figure (c) under sparse latent process. Please note that theorem is discussed
3

---

Published as a conference paper at ICLR 2025
under Markov Network. For more details, please refer to Appendix B.6 Specifically, first leverage the connection between conditional independence and cross derivatives (Lin) 1997) and the sufficient variability of temporal data to discover the relationships between the estimated and the true latent variables, which is shown in Theorem T Moreover, we establish the identifiability result of latent variables by enforcing the sparse causal influences, as shown in Theorem Moreover, we also show that the existing identifiability results of the temporally causal representation learning methods are a special case of our IDOL method, which is shown in Corollary [ mild assumption, we achieve identification of the causal graph underlying the latent causal process, as shown in Theorem
Finally, under
3.1 RELATIONSHIPS BETWEEN GROUND- TRUTH AND ESTIMATED LATENT VARIABLES
In this section, we figure out the relationships between ground truth and estimated latent variables under a temporal scenario with instantaneous dependence. In order to incorporate contextual informa- tion to aid in the identification of latent variables zt. latent variables of L consecutive timestamps including 2e, are taken into consideration. Without loss of generality, we consider a simplified case, where the length of the sequence is i.e., 2, and the time lag is 1, i.e., T 1. Please refer to Appendix B.2 for the general case of multiple lags and sequence lengths.
Theorem 1. For a series of observations X E R" and estimated latent variables 2, E R. with the corresponding process (.)(()),, where 9 is invertible, suppose that the process subject to observational equivalence Xt = ((a)). Let Ct 2 {2t-1, 2,.3 E R2nn and that Mc, be the variable set of two consecutive timestamps and the corresponding Markov network respectively. Suppose that the following assumptions hold:
Al (Smooth and Positive Density): The conditional probability function of the latent variables ct is smooth and positive, i.e,, p(Ce) (2)) is third-order differentiable and p(c, (2-)) > O over R22",
A2 (Sufficient Variability): Denote [Mell as the number of edges in Markov network Mo. Let
8 log p(c)(
Id logp(cc/zi-2)
um(m) =(
det, 2.m
de Det 02l
(5)
8 log p (2)2))
o² loggp(c
8 logp(ce(2(22)
@
dc,,10 -2.m
2.m
dc, Ott
Oc,, Ocr, 02m-2.m 2,m (1))EE(M)))
where @ denotes concatenation operation and (i,j) E E(Me,)) denotes all pairwise indice such that Cr,s. Ct. are adjacent in MM.. For m E [].... m,). there exist 4n + [Mc,l different values of Z such that the 4n + |Me,l values of vector functions w(m) are linearly independent.
Then for any two different entries εr & of c E RR20 that are not adjacent the Markov network Mà, over estimated è,, (i) Each ground-trune truth latent variable Cτ of R2an is a function of at most one of ć and én (ii) For each pair of ground-trutt latent variables C₁ and ct.j of C₁ E R that are adjacent in Mc. over Ct. they can not be a function of Ca.k and ē,, respectively.
Proof Sketch. The proof can be found in Appendix B.I First, we establish a bijective transformation between the ground-truth z; and the estimated 2, to connect them together. Next, utilize the structural properties of the ground-truth Markov network Mc, to constrain the structure of the estimated Markov network Me, through the connection between conditional independence and eross derivatives (Lin 1997), i.e., Cti 1 c.ylo J.ji implies Ø² logp(cs) "e,,000,, 0. By introducing the sufficient variability assumption, further construct a linear system with full rank coefficient matrix to ensure that the only solution holds, i.e..
dea. dea. = decs deei O, 0e, De,, de,,
Pa.
0,
(6)
06, De,
Scc. der, 0 and der, dc..1 correspond to statement (i) statement (ii), respectively. where De, de.t
de,t De,,
Theorem I provides an insight, such that when observational equivalence holds, the variables that directly related in the true Markov network must be functions of directly related variables in
4

---

Published as a conference paper at ICLR 2025
the estimated Markov network. Please note that and L can be easily generalized to any value by making some modifications on the assumption A2. The r timestamps that are conditioned on provide sufficient changes, and L sequence length provides sparse structure, which will be discussed in Theorem 2 A detailed discussion is given in Appendix B.2
3.2 IDENTIFICATION OF LATENT VARIABLES
In this subsection, we demonstrate that given further sparsity assumptions, the latent Markov network over Ct and the latent variables are also identifiable. For a better explanation of the identifiability of latent variables, we first introduce the definition of the Intimate Neighbor Set (Zhang al. 2024).
Definition 2 (Intimate Neighbor Set (Zhang et al., 2024)). Consider a Markov network Ma over variables set Z, and the intimate neighbor set of variable 2t,i is
V (,,,(e A [cr.) let, is adjacent ct., and it is also adjacent to all other neighbors of of. EcM/er.)
In other words, ! contains all neighbors, iff all neighbors of Ct2 are also in the same unique clique as Cr,s, else V = We further provide an example in the Appendix B.8
Based the conclusions of Theorem I we consider 3 consecutive timestamps, i.e., C₁ (z--1, 2t, 2,41), where the identifiability of z, can be assured with the help of the contextual information Z and Z.
Theorem 2. (Component-wiss Identification of Latent Variables with instantaneous dependencies.) Suppose that the observations are generated instantaneous latent process, and Me. the Markov network over e = {Zt-1, 1, Zt, Zt+1) E IR3", Except for the smooth, positive density and sufficient variability assumptions, we farther make the following assumption:
A3 (Sparse Latent Process ): For any Zia E Zt the intimate neighbor set of zi is an empty set.
When the observational equivalence is achieved with the minimal number of edges of the estimated Markov network of Me,, then we have the following two statements:
(i) The estimated Markov network Me, is isomorphic the ground-truth Markov network Me.-
(ii) There exists a permutaaiinn x of the estimated latent variables, such that 24 and En(iyy is one-to-one corresponding, i.e., Zut. is component-wit identifiable.
Proof Sketch. Given the fact that there always exists row permuta- tion for each invertible matrix such that the permuted diagonal entries non-zero (Zheng & Zhang 2024), we utilize the conclusion of Theorem [ to demonstrate that any edge present in the true Markov network will nec- essarily exist in the estimated one. Furthermore, when the number of es- timated edges reaches a minimum, the identifiability of Markov network can be achieved. Finally, we once again leverage Theorem [ to illus- trate that the permutation which al- lows the Markov network to be iden- tified can further lead to component- wise level identifiability of latent vari- ables. The detailed proof and discus- sion of assumptions are given in Ap- respectively.
1--10
ZeA1
Zeesa
Z
Zea
Ze=12
Ze-12
Ze.2
2
Zr,3
(a) q (Zc-1.2p Zet1)
(b) G 1444
(c) c (4)
Figure 2: Examples of Markov networks with different types of contextual information that satisfy the sparse latent pro- cess assumption. Case (i) uses both historical and future information for identifiability. Case (ii) uses only historical information. Case (iii) that does not include instantancous dependencies can identify latent variables without any con- textual information, which degenerates to TDRL (Yao et al.) 2022). Please note that all cases, we show only the Markov net of c, conditioned on previous timestamps while omitting the conditions for simplicity.
pendix B.3 and C
Discussion.
Intuitively, Theorem tells us that any latent variable 21, is identifiable when its intimate set empty, which benefits from the sparse causal influence. Compared with the existing
Please refer the definition of isomorphism of Markov networks in Appendix B.7
5

---

Published as a conference paper at ICLR 2025
Table 1: The summary of related work of causal representation learning.
No Intervention
No Grouping
Stationarity
Instantancous Effect Temporal Data
IDOL
Yao ct al. 2022
Morioka Hyvanien
2023
Lippe al.
2025
X
Zhang ct ai
2024
identifiability results for instantaneous dependencies like intervention or grouping observations, this assumption is more practical and reasonable in real-world time series datasets.
One more thing to note is that is not necessary to utilize all contextual information for identifiability. In fact, long there exist consecutive timestamps where the intimate sets of all variables in Z, empty, the identifiability can be established.
Corollary 1. (General Case for Component-wise Identification.) Suppose that the observations are generated by Equation (1) and (2), and there exists C₁ = (Zt-a:".. ,Za.... Ze+b) [R(a+b+1)xx with the corresponding Markov network Me.- Suppose assumptions Al and A2 hold true, and for any 2 E Zg. the intimate neighbor set of z,, is an empty set. When the observational equivalence is achieved with the minimal number of edges of the estimated Markov network of Me. there must exist a permutation ιt of the estimated latent variables, such that 2t,; and a)) is one-to-one corresponding, i.e,, Zz,i is component-wis wise identifiable.
Discussion. In this part, we give a further discussion on how contextual information can help to identify current latent variables. Let us take Figure (i) as an example, when we consider the current latent variables 2,,1; 2t, and 21,3 only, it is easy to see that neither 2t,1 nor 21,3 are identifiable since both of their intimate set are {2,,2] However, when the historical information Z;-1 is taken into consideration, where 1;, affects only Ft, but not 2r,,, then It, will immediately be identified since the intimate set of žt, will be empty thanks to the unique influence from 2,11,1- Similarly, with a future variable 2,+ which is influenced by 2r,3 but not 2t,1, then 21,3 can be identified as well. Besides, under some circumstances, only part of the contextual information is needed. Therefore, the length of the required time window L can be reduced as well, which leads to more relaxed requirement of assumption A2, according to Theorem | As shown in Figure 2 (ii), with only historical information, the identifiability of 2 can already be achieved. Furthermore, some existing identifiability results (Yao et al. 2021 2022) can be considered as a special case of our theoretical results when the instantaneous effects vanish. For example, Figure 2 (iii) provides the example of Yao et al. 2022). where all intimate sets are empty naturally, and the identifiability is assured.
TDRL
Finally, we explore under what circumstances the causal graph of latent variables is also identifiable.
Theorem 3. (Identification of Latent Causal Process.) Suppose that the observations are generated by Equation (12)). and that Me. is the Markov network over C₁ (Zt-1, 2e} R²n Suppose that all assumptions for Theorem 2 hold. We further make the following assumption: for any pair of adjacent latent variables 21,1, 2tj at time step t, their time-delayed parents are not identical, i.e., Pa)(z).)) F Pas(z1,)) Then, the causal graph of the latent causal process is identifiable
Proof Sketch. We first establish the identification of the skeleton of the causal graph using the results of Theorem 2 The directions of time-delayed edges are fully determined since time can only move forward. Finally, we leverage the v-structure to identify the instantancous edges. A detailed proof is provided in Appendix B.5
3.3 COMPARASION WITH EXISTING METHODS
The application of real-world scenarios presents numerous new opportunities and challenges. Com- pared to existing methods of temporally causal representation learning with instantaneous effects, our approach significantly relaxes the constraints on theoretical assumptions, as shown in Table [ TDRL Yao et al. 2022 requires that there are no instantaneous effects during the transition process, which can be considered as a special case of our method. G-CaRL (Morioka & Hyvärinen 2023) and iCITRIS (Lippe et al. 2023) require extra intervention grouping information, which is not neces- sary in our method. Zhang et al. (2024) focus on non non-temporal scenarios and require extra domain labels. Instead, IDOL can handle the stationary temporal case. Besides, when the instantaneous effect

---

Published as a conference paper at ICLR 2025
is dense, which is beyond the capacity of Zhang et al. (2024). IDOL can still utilize the contextual information for better identifiability as long as the temporal transition is sparse.
IDENTIFIABLE INSTANTANEOUS LATENT DYNAMIC MODEL
4.1 TEMPORALLY VARIATIONAL INFERENCE ARCHITECTURE
Based on the data generation process, we derive the evidence lower bound (ELBO) as shown in Equation 7 for general time series modeling. Please refer to Appendix E.2 for the derivation details of the EBLO for time series forecasting.
Prior Network r2 2;-1)
Lx
his
{2, 2z "n, ,273
ELBO = ( In Inp(xir|
Encoder
Decoder
by
[x X2, ,,)
(8 82. 1 8y.
-DKI(e(Zir XL.) Xx.) (P((T))
The framework of the IDOL model. The encoder and decoder are used for the extraction of latent variables and ob- servation reconstruction. The prior network is used for prior distribution estimation, and Lg denotes the gradient-based sparsity penalty. The solid and dashed arrows de- note the forward and backward propagation.
Figure 3:
where DKZ denotes the KL divergence. We use pos- terior q(Z1:r (x1:1) to approximate the prior p( p(Z1:r). Besides, p( X1: (i:r) is used to reconstruct the obser- vations. The encoder and the decoder can be formalized as follows:
1 ZET +(xir)
((():)),
(8)
where @ and 3 denote the neural architecture based on convolution neural networks. Please refer to Table A4 for the implementation details of á and tb
Based on thcoretical results, we develop the IDOL model as shown in Figure variational inference to model the distribution of observations. To estimate the prior distribution and enforce the independent noise assumption, we devise the prior networks. Moreover, we employ a gradient-based sparsity penalty promote the sparse causal influence.
3 which is built on the
4.2 PRIOR ESTIMATION
To enforce the independence of noise in Equation we minimize the Kullback-Leibler (KL) divergence between the posterior distribution 11, p( 2 (a,((aa,,,))) (a. xt) and a prior II. (Pa((())) (2t,)) This minimization implies that latent variables are mutually inde- pendent and conditioned on their historical and instantaneous parents. However, since the prior distribution may have any arbitrary density function, it is difficult to estimate such a prior. To solve this challenge, follow Chen al. (2024a), Yao al. (2022) and propose the prior networks. Specifically, we first let r; be an inversed function of / from Equation (2) that are implemented by normalizing flow. Papamakarios et al.) (2021) take the estimated latent variables as input estimate the noise term é;, i.c. &,, = (2, 2, - And each r; is implemented by Multi-layer Perceptron networks (MLPs). Sequentially, we devise transformation K = {2,-1, 2,} (21-1, 2,), whose
0 Jacobian can be formalized J, Ja Je
dr: and J. 0r, where Ja 86c-1.1)
Hence
(8h..
we have Equation (9) via the change of variables formula.
Or:
log log, 2 log p(2 log Zz,,1
(9)
According to the generation process, the noise é,, is independent with 2,-1, so we can enforce the independence of the estimated noise term êt,,. And Equation (9) can be further rewritten as
T
Or:
logp(?ir) log +
Σ poe,() + Σ log
(10)
a=c.
where p(ê-,) is assumed to follow a Gaussian distribution. Please refer to Appendix E.1 for more details of the prior derivation.
We use the superscript symbol denote estimated latent variables
7

---

Published as a conference paper at ICLR 2025
4.3 GRADIENT-BASE BASED SPARSITY REGULARIZATION
Ideally, the MLPs-based architecture r; can capture the causal structure of latent variables by restrict- ing the independence of noise é,,i. However, without any further constraint, r; may bring redundant causal edges from 2 [e.[n]]d to 2, leading to the incorrect estimation of prior distribution and further the suboptimization of ELBO. As mentioned in Subsection 4.2, Ja and Jg intuitively denote the time-delayed and instantancous causal structures of latent variables, since they describe how the
2, 2x.[ [ contribute to 21,5, which motivate us to remove these redundant causal edges with a sparsity regularization term Ls by simply applying the L1 on Ja and Je. Formally, we have
(11) where || * || denotes the Ci Norm of matrix. By employing the gradient-based sparsity penalty on the estimated latent causal processes, we can indirectly restrict the sparsity of Markov networks to satisfy the sparse latent process. Finally, the total loss of the IDLL model can be formalized as:
Ls = || ||
Liotal Cr aLk BLs.
(12)
where a, denote the hyper- parameerass.sss...
5
EXPERIMENTS
5.1
EXPERIMENTS ON SIMULATION DATA
5.1.1
EXPERIMENTAL SETUP
Data Generation. We generate the simulated time series data with the fixed latent causal process as introduced in Equations (1)-(2) and Figure (c). To better evaluate the proposed theoretical results, we provide six synthetic datasets from A to with 3, 5, 8, 8, 8, 16 latent variables, respectively. Dataset D contains instantaneous effects, which degenerate the TDRL (Yao et al. 2022) setting as introduced in Figure (iii). Dataset E has dense latent causal process that violates the assumption A3(Latent Process Sparsity). All datasets except E satisfy the assumption A3. Please refer to Appendix F.I.I for the details of data generation and evaluation metrics.
Baselines. To evaluate the effectiveness of our method, consider the following compared methods. First, we consider the standard /3- VAE (Higgins et al. 2017) and Factor VAF (Kim & Mnih 2018). which ignores historical information and auxiliary variables. Moreover, we consider TDRL (Yao
et al. 20222. iVAE (Khemakhem et al. (2020a). TCL (Hyvarinen & Moriokal 2016), PCL (Hyarinnn
Morioka 2017). and Slow VAE (Klindt al. 2020). which use temporal information but do not assume instantaneous dependency on latent processes. Finally, we consider the iCITRIS (Lippe et al.) 2023) and G-CaRL (Morioka Hyvärinen] 2023), which are devised for latent causal processes with instantaneous dependencies but require interventions grouping of observations. As for G-CaRL, we follow the implementation description the original paper and assign random grouping for observations. As for iCITRIS, assign random intervention variable for latent variables. We repeat each method over three random seeds and report the average results.
5.1.2 RESULTS AND DISCUSSION
Experiment results of the simulation datasets are shown in Table 2 The proposed IDOL model achieves the highest MCC performance, reflecting that our method can identify the latent variables under the temporally latent process with instantaneous dependency. According to the experiment results, we can obtain the following conclusions: 1) our IDOL method also outperforms G-CaRL and iCITRIS, which leverage grouping and interventions for identifiability of latent variables with instantaneous dependencies, reflecting that our method does not require stricter assumption for identifiability. 2) Compared with the existing methods for temporal data like TDRL, PCL, and TCL, the experiment results show that method can identify the general stationary latent process with instantaneous dependencies. 3) According to the results of the dense dataset E, the IDOL cannot well identify the latent variables, validating the boundary of our theoretical results. 4) We further analyze the effectiveness of the sparsity regularization. Please refer to Appendix F224 for more details.
Quantitative Results:
Qualitative Results: Figure provides visualization results on dataset A the latent causal process of IDOL,, TDRL, G-CaRI., and iCITRIS. Note that the visualization results of our method are
8

---

Published as a conference paper at ICLR 2025
Table 2: Experiments results of MCC simulation data. The iCITRIS algorithm encountered
out-of-memory (OOM) issue when running on dataset F due to the high dimensionality of the latent
variables, which caused the code fail to execute. This is indicated by the symbol '-'.
Datasets
IDOL
TDRL
G-CaRL
iCITRIS
VAE
Slow VAE
iVAE
Factor VAE
PCL
TCL
0.9645
0.9416
0.9059
0.8219
0.8485
0.8512
0.6283
0.8512
0.8659
0.8625
B
0.9142
0.8727
0.6248
0,4120
0.4113
0.2875
0.5545
0.2158
0.5288
0.3311
0.9801
0.9001
0.5850
0.4234
0.4093
0.3420
0.6736
0.2417
0.3981
0.2796
0.9766
0.9796
0.5455
0.3343
0.2181
0.2641
0.3469
0.2527
0,4806
0.2461
E
0.7869
0.7228
0,5835
0,4646
0.4260
0.3986
0.6071
0.2319
0,4659
0.2881
0.9747
0.5899
0.5225
0.4321
0.5157
0.3176
0.2648
0.5908
0.6678
Ground Truth
IDOL
TDRL
G-CaRL
iCITRIS
Time delayed
2.42
0.35
0.16
3.36
1.4 0.98
4.15
1.12
3.2
1.0
1.0
1.0
To
TO
0.18
0.79
0.08
10
1.36
6.48
1.01
P
0.32
1.85
2.01
1.0
1.0
1.0
2,3
0.04
0.11
0.54
0.88
2.55
1.75
0.53
2.05
4.92
1.0
1.0
1.0
Instantaneous
Z11
x:1
0.0
0.0
0.0
0.0
0.0
0.0
2.1
0.0
0.0
0.0
0.0
0.0
0.0
TO
TO
1.47
0.0
0.0
0.0
0.0
0.0
P
1.48
0.0
0.0
0.5
0.0
0.0
2.3
0.15
0.35
0.0
0.0
0.0
0.0
1.6
0.94
0.0
0.5
0.5
0.0
2;11
Z;,2
X;33
Figure 4: Visualization results of directed acyclic graphs of latent variables of different methods. The first and second rows denote time-delayed and instantaneous causal relationships of latent variables.
generated from Ja and Jr. According to the experiment results, we can find that the proposed method can reconstruct
latent causal relationship well, which validates the theoretical results. Please note that here, not only the Markov equivalence class but also the causal graph can be identified for dataset A, as shown in Figure 4 Please refer to Appendix B.5 for more details. Moreover, since TDRL employs the conditional independent assumption, it does not reconstruct any instantancous causal relationships. Besides, without any accurate grouping or intervention information, the G-CaRI. and iCITRIS can not reconstruct the correct latent causal relationships. Please refer to Appendix E.3. for more details.
5.2
EXPERIMENTS ON REAL-WORLE DATA
5.2.1 EXPERIMENT SETUP
Datasets. To evaluate the effectiveness of our IDOL method in real-world scenarios, we conduct experiments two human motion datasets: Human3.6N (Tonescu al. 2014) and HumanEva-1 (Sigal et al. 2010), which record the location and orientation of local coordinate systems at each joint. We consider these datasets since the joints can be considered as latent variables and they naturally contain instantaneous dependencies. We choose several motions to conduct time series forecasting. For the Human3. 6M datasets, we consider motions: Gestures (Ge), Jog (J), Catch (CT), and Walking (W). For HumanEva-1 dataset, we consider 6 motions: Discussion (D), Greeting (Gr), Purchases (P), SittingDown (SD), Walking (W). and WalkTogether (WT). For each dataset, we select several motions and partition them into training, validation, and test sets. Please refer the Appendix F.211 for the dataset descriptions.
Baselines. To evaluate the effectiveness of the proposed IDOL,, we consider the following state- of-the-art deep forecasting models for time series forecasting. First, we consider the conventional methods for time series forecasting including Autoformer (Wu et al. 2021), TimesNet (Wu et al.) 2022) and MICN (Wang et al.) 2022). Moreover, we consider several latest methods for time series analysis like CARD (Wang et al. 2023). FITS (Xu et al. 2024b), and iTransformer (Liu et al. 2024). Finally, we consider the TDRL Yao et al. 2022). We repeat each experiment over 3 random seeds and publish the average performance. Please refer to Appendix F22 for more experiment details.

---

Published as a conference paper at ICLR 2025
Table 3: MSE and MAE results on the different motions.
dbteect
Predict
IDOL
TDKL
CARD
FTIS
MICN
iirasfoomr
TimesNer
Autofiommr
Length
MSE
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAA
MSE
MAE
MSE
MAE
MSE
MAB
0,0688
6.1623
0.079
0.1806
0.0898
0.199
0,0738
0.1788
0.781
0.1896
0.0605
0.2013
0.1949
0.109
0.323
0,001
0.1916
00878
0.1991
0.0996
0.1955
0.834
0,R12
0.1990
0.0801
0.2000
0.060
0.1934
01000
0.3195
0,8697
0.1754
00724
0.1796
0.0627
0.184
0.0866
0.1990
0.715
0.178
0.0875
0.165
0.631
0.1942
0.00
0.2192
Huntareo
0.1516
02310
0.3392
0.336
0.3466
0.3277
0.4055
0.15%
0.2557
0.323
0.2189
0,3071
0.2370
0.2532
0.3633
0
0.441
0.3614
0.3936
0.3396
0.4142
0.1648
0.2713
0.2874
0.2095
0.3526
0.3367
0.3199
0.1742
0.2786
0.8110
0,3110
0.3938
0:4266
0.3552
0.4329
0.1864
0.2945
0,3074
0.3841
0,3056
0.5707
0.3447
0.2934
0.0066
0.0166
0.0651
0.069
0,0147
00054
0.1049
00151
00611
0.991
0.0142
0.0635
0.065
0.1062
0.0175
0.191
0,0966
0,028
0,0636
0.0166
0,0729
0.0857
0.0869
0.0299
0.1108
0.0700
6,1338
0.988
0,174
0,1010
0.1641
0.103
0.3117
0.0967
0.1851
0,8604
0.444
0,005
0.170
0.040
0.1767
W
0.1183
6.1814
0.146
03172
0,3387
0.25%
0.31
0,3889
01111
6:222
0.1245
0.2943
0.1750
0.239
0.3993
0.3498
75
0.1977
0.2545
0.2091
03642
0.4777
0.3675
0.3156
0.3817
0.1124
0.27%6
0.2339
0.2784
0.2200
0.2884
0.3654
0,4009
125
0.0071
6,0885
0,0074
0,50
0,089
00550
0,0085
010000
0:0524
0,0076
0.036
0.0097
0,0565
0.0004
0,0586
250
0,0094
0J563
0.0966
0,890
00117
00000
00011
0,0999
00067
0:096
0.0112
00581
0.013
0,043
00134
0,0056
375
0,0102
0.0638
00106
0,001
00138
00644
0,0034
0,0555
0,0009
0,8608
0.0126
0.817
0,0122
0,0615
00141
0.0678
125
0.0129
0.0641
00167
0
0.099
0,0239
0.8665
0014
0
0.0137
0.6649
1.0105
0,078
00217
0,0845
250
0,0158
00218
0,890
0
0,0088
0.082
D.847
0.0217
0.832
0.0277
03533
00287
0.0974
375
0.0226
0,0902
00234
0,914
0.0255
0.0304
0.1011
011233
0.0912
0.0263
0.0920
0A311
0.098
00319
0.1029
Hunus
0.82%6
0.7788
0.1003
00233
00303
6:866
6100
0/004
OIHIT
0.837
0.109
0.626
0,027
0-1164
0
00227
0,0862
0:3146
0.0225
0.0434
0,793
0.1182
0,006
0,554
02139
01337
00546
00400
0.1106
0.336
0.8324
0.1104
0.033
63148
0,0400
0.1288
0,000
0.3315
011364
0.319
0.0495
0.1312
0.0595
0.1439
0,0338
0.1498
00157
0,077
0.0155
00772
0,026
0.0889
0,0162
0.279
0,00
0,0333
0.113
0,031
0,1175
0,1212
00V8
0,030
0.1186
00102
0,0590
0.0113
0.6666
0.0444
0.0941
0.0612
0.0123
0.0610
0.0622
0.0728
00156
0,0729
0.051
0.1033
0,2995
0.144]
0,0364
0.1947
0.1381
0.8193
0.07%8
00007
0,099
0.0420
0.1588
0.1135
0.1548
0.206
0.804
0.0483
0.176
0.0455
0. 1080
0.131
0.1617
125
0.8129
0.1667
0.0155
0.0676
0.0195
0.0779
0
0.124
0.1155
0.063
0.0180
0.0719
0.0216
0,0787
0.0735
0.1301
WT
250
0.8206
0.0515
0.0277
0.0630
0.3149
0.1147
0.1127
0.1623
011211
0.821
0.0437
0.1883
110425
0.1658
0.1041
0.1366
375
0,8233
0.0575
0.0248
0.0856
0.0552
0.1255
0.3149
0.1441
0U1245
0.2857
0.0474
0.1146
0,0456
0.1122
0.1165
0.3654
5.2.2 RESULTS AND DISCUSSION
Experiment results of the roal-world datasets are shown in Table 3 We also conduct the Wilcoxon filloxonniignn signed-rank test Richard (2021) on the reported accuracies, our method significantly outperforms the baselines, with a p-value threshold of 0.05. According to the experiment results, our IDOL model significantly outperforms all other baselines on most of the human motion forecasting tasks. Specifically, our method outperforms the most competitive baseline by a clear margin of 4%-34% and promotes the forecasting accuracy substantially on complex motions like the Sitting Down (SD) and the Walking (W). This is because the human motion datasets contain more complex patterns described by stable causal structures, and our method can learn the latent causal process with identifiability guarantees. It is noted that our method achieves a better performance than that of TDRL, which does not consider the instantancous dependencies among latent variables. We also find that our model achieves a comparable performance in the motion of Catch Throw (CT), this is because the size of this dataset is too small for the model to model the distribution of the observations. We further consider a more complex synthetic mixture of human motion forecasting, experiment results are shown in Appendix F.2.2 Please refer to Appendix [-2.4 for the experiment results of ablation studies. Moreover, we also investigate the proposed IDOL model on other high- digeeniinnnoon datasets in Appendix H.T
Quantitative Results:
6 CONCLUSION
This paper proposes general framework for time series data with instantaneous dependencies to identify the latent variables and latent causal relations up the Markov equivalence class. Further- more, with mild assumption, the causal graph is also identifiable. Different from existing methods that require the assumptions of grouping observations and interventions, the proposed IDOL model employs the sparse latent process assumption, which is easy to satisfy in real-world time series data. We also devise a periational-iffeenne--e based method with sparsity regularization to build the gap between theories and practice. Experiment results on simulation datasets evaluate the effectiveness of latent variables identification and latent directed acyelic graphs reconstruction. Evaluation in human motion datasets with instantaneous dependencies reflects the practicability in real-world scenarios. There are two main limitations in our work. First, our method is not for high-dimensional time series data because modeling the complex dependencies through sparsity constraints is challenging. Second, our method relies on the invertible mixing process, but it may not hold in real-world scenarios. How to address these limitations and make our method more scalable will be interesting direction.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
We would like to acknowledge the support from NSF Award No. 2229881, AI Institute for Societal De- cision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. Moreover, this research was supported part by the National Science and Technology Major Project (1212D0011501)) Natural Science Foundation of China (U24A20233).
REFERENCES
Sudeshna Adak. Time-dependent spectral analysis of nonstationary time series. Stanford University, 1996.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01221, 2018.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(7970):533-555 2023.
George EP Box and David Pierce. Distribution of residual autocorrelations in autoregressive- integrated moving average time series models. Journal of the American statistical Association, 65 133-):1509-1526, 1970.
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019.
Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangehen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. arXiv preprint arXiv:2401. 14535, 2024a.
Wei Chen, Zhiyi Huang, Ruichu Cai, Zhifeng Hao, and Kun Zhang. Identification of causal structure with latent variables based on higher order cumulants. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20353-20361, 2024b.
Yuxiao Cheng, Lianglong Li, Tingxiong Xiao, Zongren Li, Jinli Suo, Kunlun He, and Qionghai Dai. Cuts+: High-dimensional causal discovery from irregular time-series. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 11525-11533. 2024.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
W Gersch. Modeling nonstationary time series and inferring instantaneous dependency, feedback and causality: application to human epileptic seizure event data. IFAC Proceedings Volumes, 18(5):737-742, 1985.
Mingming Gong, Kun Zhang, Bernhard Schoelkopf, Dacheng Tao, and Philipp Geiger. Discovering temporal causal relations from subsampled data, In International Conference on Machine Learning. pp. 1898-1906. PMLR, 2015.
Mingming Gong, Kun Zhang, Bernhard Schölkopf, Clark Glymour, and Dacheng Tao. Causal discov- ery from temporally aggregated time series. In Uncertainty in artificial intelligence: proceedings of the... conference. Conference on Uncertainty Artificial Intelligence, volume 2017. NIH Public Access, 2017.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint urXiv:2111.0000 2021a.

---

Published as a conference paper at ICLR 2025
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585. 2021b.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971- 1- 35983, 2022.
Hermanni aa and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pp. 939-948. PMLR, 2020.
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongje Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34: 10216644-6333 1633, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
17850 1997.
Yingyao Hu and Susamne M Schennach. Instrumental variable treatment of nonclassical measurement error models. Econometrica, 76(1):195-216, 2008
Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 35:5549-5560 -5511, 2022.
Zenan Huang, Haobo Wang, Junbo Zhao, and Nenggan Zheng. Latent processes identification from multi-view time series. arXiv preprint arXiv:2305.08166 2023.
Dat Huynh, Elizabeth Fons, and Svitlana Vyetrenko. Few-shot learnable augmentation for financial time series prediction under distribution shifts. In NeurlPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Aapo Hyvärinen. Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Enginecring Sciences, 371(19844:20110534, 2013.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks. ((4-5):411-430 2000.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 4399 1999.
Aapo Hy Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of a structural vector autoregression model using non-gaussianity. Journal of Machine Learning Research, 11(5), 2010.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of varialle and structural-equation models: from linear nonlinear. arXiv preprint arXiv:2302.02672 2023.
12

---

Published as a conference paper at ICLR 2025
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(7):1325-133) -333999,, 2014. doi: 10. 10.1T09TPPMI. 2013.248.
Noor Jamaludeen, Vishnu Unnikrishnan, André Brechmann, and Myra Spiliopoulou. Discovering instantaneous granger causalities in non-stationary categorical time series data. International Conference on Artificial Intelligence in Medicine, pp. 200-209. Springer, 2022.
Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang, and Shirui Pan. Multivariate time series forecasting with dynamic graph neural odes. IEEE Transactions on Knowledge and Data Engineering, 2022.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 13:18671-22778 2020b.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International conference on machine learning, pp. 2649-2658. PMLR, 2018.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement natural data with temporal sparse coding. arXiv preprint arXiv:2007.100 10930, 2020.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. International conference on machine learning, pp. 11455-11472. PMLR, 2022.
Lingjing Kong, Biwei Huang, Feng Xie, Eric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. arXiv preprint arXiv:2306.07916 2023.
Lingjing Kong, Biwei Huang, Feng Xie, Fric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 36, 2024.
Yu Kong and Yun Fu. Human action recognition and prediction: A survey. International Journal of Computer Vision, 13666)11366-1401 2022.
Christos Koutlis, Vasilios K Kimiskidis, and Dimitris Kugiumtzis. Identification of hidden sources by estimating instantaneous causality high-dimensional biomedical time series. International Journal of Neural Systems, 29(04): 1850051, 2019.
Sébastien Lachapelle and Simon Lacoste-Julien. Partial disentanglement via mechanism sparsity. arXiv preprint w21.::222200773333 2022.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pp. 428-484, PMLR, 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Jull Jacisee-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: General- ization and identifiability in multi-task learning. In International Conference on Machine Learning. pp. 18171-11 1811118206. PMLR, 2023.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint arXiv:2401.0423 004990 2024a.
13

---

Published as a conference paper at ICLR 2025
Sébastien Lachapelle, Divyat Mahajan, loannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian hartssianprrouut extrapolation. Advances in Neural Information Processing Systems, 36, 2024b.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 954104, 2018.
Te-Won Lee and Te-Won Lee. Independent component analysis. Springer, 1998.
Bing Li, Wei Cui, Le Zhang, Ce Zhu, Wei Wang, Ivor Tsang, and Joey Tianyi Zhou. Difformer: Multi-resolutional differencing transformer with dynamic ranging for time series analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Zijian Li, Ruichu Cai, Tom ZJ Fu, Zhifeng Hao, and Kun Zhang. Transferable time-series forecasting under causal conditional shift. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Zijian Li, Zunhong Xu, Ruichu Cai, Zhenhui Yang, Yuguang Yan, Zhifeng Hao, Guangyi Chen, and Kun Zhang. Identifying semantic component for robust molecular property prediction. arXiv preprint arXiv:231 .04837, 2023c.
Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. Subspace identification for multi-source domain adaptation. Advances in Neural Information Processing Systems, 36, 2024.
Juan Lin. Factorizing multivariate function classes. Advances in neural information processing systems, 10, 1997.
Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. -3557-13603. PMLR, 2022.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centtic causal representation learning. 2nd Conference on Causal Learning and Reasoning (CLeaR), 2023.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research, 2 21(209):1 2020.
Romain Lopez, Jan-Christian Hütter, Jonathan Pritchard, and Aviv Regev. Large-scale differentiable causal discovery of factor graphs. Advances in Neural Information Processing Systems, 35:992 19303, 2022.
Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. The Twelfth International Conference on Leamiing Representations, 2024.
Amin Mansouri, Jason Hartford, Yan Zhang, and Yoshua Bengio. Object-centric architectures enable efficient causal representation learning. arXiv preprint urXiv:2310.19054 2023.
Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärinen. Causal discovery with general non-line -ineirner relationships using non- non-linear ica. In Uncertainty artificial intelligence, pp. 186-195. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by grouping of observational variables. arXiv preprint arXiv:2310.15709, 2023.
Leland Gerson Neuberg. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, (KK4):675-685. 2003.
Ignavier Ng. Yujia Zheng, Xinshuai Dong, and Kun Zhang. On the identifiability of sparse ica without assuming non- non-gaussianity. Advances in Neural Information Processing Systems, 36, 2024.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.147300 2022.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 1 1-64, 2021.
Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradcep Ravikumar. Learning interpretable concepts: Unifying causal representation learning and foundation models. arXiv preprint arXiv:2402.09236, 2024.
Lowry Richard. Concepts and applications of inferential statistics, 2014, 2021.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1181-1191, 2020.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)6612-65) 634, 2021.
Leonid Sigal, Alexandru O. Balan, and Michael J. Black. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. J. Comput. Vis., 87(1-2):44222 2010. doi: 10.1007/S1 263-009-0227-6
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001.
Norman Swanson and Clive WJ Granger. Impulse response functions based on causal ap- proach to residual orthogonalization in vector autoregressions, Journal of the American Statistical Association, pp. 357-367, 1997.
Binh Tang and David Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592 23608, 2021.
Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily B Fox. Neural granger causality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4267-42799 2021.
Tatsuya Tashiro, Shohei Shimizu, Aapo Hyvärinen, and Takashi Washio. Estimation of causal orders in a linear non-gausse non-gaussian acyelic model: method robust against latent confounders. In Artificial Neural Networks and Machine Learning- ICANN 2012: 22nd International Conference
Artificial Neural Networks, Lausanne, Switzerland, September 11-14, 2012, Proceedings, Part
22, pp. 491-494 498-498. Springer, 2012.
Ruey S Tsay. Analysis of financial time series. John wiley & sons, 2005.
Haishuai Wang, Qin Zhang, Jia Wu, Shirui Pan, and Yixin Chen. Time series feature learning with labeled and unlabeled data. Pattern Recognition, 89:55 66.5566,770 2019.
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference Learning Representations, 2022.
Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. The Twelfth International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2025
Liang Wendong, Armin Kekić, Julius Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. Advances in Neural Information Processing Systems, 36, 2024.
Anpeng Wu, Haoxuan Li, Kun Kuang, Keli Zhang, and Fei Wu. Hierarchical topological ordering with conditional independence test for limited time series. arXiv preprint arXiv:2308. 148. 2023a.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 144-22419-224360 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations, 2022.
Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nature Machine Intelligence, 5(6):602-611, 1, 2023b.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng. and Kun Zhang. Identification of linear non non-gaussian latent hierarchical structure. In International Conference on Machine Learning. pp. 24370-24387 PMLR, 2022a.
Shaoan Xie, Lingjig Kong, Mingming Gong, and Kun Zhang, Multi-domain image generation and translation with identifiability guarantees. In The Eleventh International Conference on Learning Representations, 2022b.
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. arXiv preprint arXiv:2403.08333, 2024a.
Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: Modeling time series with $10k$ parameters. In The Twelfth International Conference on Learning Representations, 2024b.
Hanqi Yan, Lingjing Kong, Lin Gui, Yujie Chi, Eric Xing, Yulan He, and Kun Zhang. Counterfactual generation with identifiability guarantees. In 37th International Conference on Neural Information Processing Systems, NeurlPS 2023, 2023.
Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Ijcai, volume 15, pp. 3995-4001. Buenos Aires, Argentina, 2015.
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. arXiv preprint S..Xx:2311.0000:: 2023.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint 1rX0v:2110.05422, 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. Advances in Neural Information Processing Systems, 35:26492 266503, 2022.
Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, et al. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv preprint arXiv:2306.1011 2023.
Kun Zhang and Laiwan Chan. Kernel-based nonlinear independent component analysis. In International Conference on Independent Component Analysis and Signal Separation, pp. 301- 308. Springer, 2007.
Kun Zhang and Aapo Hyvarinen. On the identifiability of the nont-nonl causal model. arXiv preprint aixi 105. 2012.
16

---

Published as a conference paper at ICLR 2025
Kun Zhang, Shaoan Xie, Ignavier Ng. and Yujia Zheng. Causal representation learning from multiple distributions: A general setting. arXiv preprint arXiv:2402.05052, 2024.
Min Zheng and Samantha Kleinberg. Using domain knowledge to overcome latent variables in causal inference from time series. In Machine Learning for Healthcare Conference, pp. 474-489. PMLR, 2019.
Yujia Zheng and Kun Zhang. Generalizing nonlinear ica beyond structural sparsity. Advances in Neural Information Processing Systems, 36, 2024.
Yujia Zheng, Ignavier Ng. and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in Neural Information Processing Systems,
2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Zhengmao Zhu, Yuren Liu, Honglong Tian, Yang Yu, and Kun Zhang. Beware of instantaneous dependence in reinforcement learning. arXiv preprint arXiv:2303 055558, 2023.
17

---

Published as a conference paper at ICLR 2025
Supplement to
"On the Identification of Temporally Causal Representation with Instantaneous Dependence"
Appendix organization:
Related Works
20
A.1
Identifiability of Causal Representation Learning
20
Nonlinear ICA for Time Series Data
20
A.3
Causal Discovery with Latent Variables
20
Instantancous Dependencey of Time Series Data
A.5
Time Series Forecasting
21
B
Proof of Theory
21
B.1
Relationships between Ground -hound-truuth and Estimated Latent Variables
B.2
Extension Multiple Lags and Sequence Lengths
24
B.3
Identifiablity of Latent Variables
24
B.4
General Case for Component-wise Identifications
25
B.5
Identifiability of Latent Causal Process
26
Markoy Network
27
B.7
Isomorphism of Markov networks
28
B.8
Illustration of Intimate Neighbor Set
28
Discussion of Assumptions
28
D The importance of considering instantaneous dependence
29
Implementation Details
29
Prior Likelihood Derivation
29
Evident Lower Bound
................................
30
Model Details
30
E.3.1
Reproducibility of Simulation Experiments,
E.3.2
Reproducibility of Real-world Experiments,
31
Experiment Details
34
F.1
Simulation Experiment
34
Data Generation Process
34
2
Evaluation Metries,
34
E1.3
More Simulation Experiment Results
34
11.4
Ablation Study
34
18

---

Published as a conference paper at ICLR 2025
F.2
Real-world Experiments'
36
F2.1
Real-world Dataset Description
36
F22.2
More Experiments Results
36
F2.3
Qualitative Results
36
F22.4
Ablation Study
37
Complexity Analysis
37
G.1
Wall-clock Training Times
37
G.2
Model Efficiency Analysis
38
More Discussions
38
Analysis of High-Diiensioon Data
38
H.1.1
Simulation Experiments)
................................
38
H.1.2
Real-world Experiments
39
H.I.3
Potential Solution for High Dimension Data
39
H.2
Noisy Environments
39
Analysis of Sample Size
40
Subsampled Time Series
40
19

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
A.1 IDENTIFIABILITY OF CAUSAL REPRESENTATION LEARNING
To achieve the causal representation (Rajendran al. 2024 Mansouri al. 2023 Wendong et al..
2024) for time series data, several researchers leverage the independent component analysis (ICA) to
recover the latent variables with identification guarantees (Yao et al. 2023 Schölkopf al., 202
Liu et al. 2023 Gresele et al. 2020). Conventional methods assume a linear mixing function from
the latent variables to the observed variables (Comon 1994, Hyvärinen 2013 Lee & Lee 1998
Zhang & Chan 2007). To relax the linear assumption, researchers achieve the identifiability of latent variables via nonlinear ICA by using different types of assumptions like auxiliary variables
2022 Hyvärinen & Pajunen 1999, [Hyvärinen et al., 2023 Khemakhem et al., 2020b; Li ct al.
or sparse generation process (Zheng et al.,
2023c). As for the methods that employ the auxiliary variables,KKhemakknn Khemakhhemmmmmmmmmmmmeeea et al. (2020a) first achieve the identifiability by assuming the latent sources with exponential family and introducing auxiliary variables c.g., domain indexes, time indexes, and class labels (Khemakhem et al. 2020al Hyvarinen & Morioka 2016 2017 HHyarimen et al.) (2019).
To further relax the exponential family assumption, Kong et al. (002)): Xie et al. (2022b); Kong
al. (2023); Yan et al. (2023): Xie et al. (2022b) achieve the component
identification results
for nonlinear ICA with a 2n number of auxiliary variables for n. latent variables. Recently, Li
al. (2024) further relax to n number of auxiliary variables and achieve subspace identifiability. To seek identifiability in an unsupervised manner, researchers employ the assumption of structural
sparsity to achieve identifiability (Ng et al., 2024 Lachapelle et al.) 2022 /Zheng et al., 2022, Xu
et al. 2024a). Specifically, [Lachapelle et al. (2023) : Lachapelle & Lacoste-Julin
(2022) proposed
mechanism sparsity regularization as an inductive bias to identify the causal latent factors and
achieve the identifiability on several scenarios like multi-task learning. They further show the
identifiability results up to the consistency relationship (Lachapelle et al. 2024a), which allows the partial disentanglement of latent variables. Recently, (Zhang al.) (2024) use the sparse structures of latent variables to achieve identifiability under distribution shift. Different from these methods that assume sparsity exists in the generation process between the latent source and the observation, we assume the sparse latent process, where sparsity exists in the transition of latent variables and is common in real-world time series data.
A.2 NONLINEAR ICA FOR TIME SERIES DATA
Our work is also related to the temporally causal representation. Existing methods for temporally causal representation (Yan et al. 2023 Huang et al., 2023: H'alv'a a & Hyvarinen 2020 Lippe
et al. 2022) usually rely on the conditional independence assumption, where the latent variables are mutually independent conditional on their time-delayed parents. Specifically, [Hyvarinen Morioka (2016) leverage the independent sources principle and the variability of data segments to achieve identifiability on nonstationary time series data. They further use permutation
contrastive
(Hyvarinen & Morioka 2017) to achieve identifiability stationary time series data. The recent advancements in temporally causal representation include LEAP Yaoo et al.) 2021). TDRL ( YYo et al. 2022), which leverage the properties of independent noises and variability historical informa- tion. However, this assumption is too strong to meet in weal-world scenarios since instantaneous dependencies are common in real-world scenarios like human motion. To solve this problem, re- searchers introduce further assumptions. For example, Lippe et al. (2023) propose the i-Citris, which demonstrates identifiability by assuming interventions on latent variables. In addition, Morioka & Hyvärinen (202)) propose G-CaRL, which yields identifiability with the assumption of the grouping of observational variables. However, these assumptions are still hard to meet in practice. Hence we propose more relaxed assumption, transition sparsity, to achieve identifiability under time series data with instantaneous
A.3 CAUSAL DISCOVERY WITH LATENT VARIABLES
Several studies are proposed to discover causally related latent variables (Chen et al.] 2024b
Lachapelle et al. 2024b; Monti et al.] 2020 Zhang & Hyvarinen] 2012] Tashiro et al. 2012). Specifically, (Huang et al., 2022; Kong et al. 2024 leverage the vanishing Tetrad conditions or
rank constraints to identify latent variables in linear- GrasssGa models. Xie et al. (2022a); Cai et al.
(2019) further draw upon -ou--aassiantty in their analysis for linear, non-Gaussian scenarios. Other
20

---

Published as a conference paper at ICLR 2025
methods aim to reconstruct the hierarchical structures of the latent variables, like Xie et al. (2022a); Huang et al. (2022). However, these methods usually use the linear assumption and can hardly handle the real-world time series data with complex nonlinear relationships.
A.4 INSTANTANEOUS DEPENDENCEY TIME SERIES DATA
When the discrete time series data is sampled in low- low-frequency or low-resolution manner, instanta- neous dependence (Adak) 1996 Koutlis et al. 2019 Gersch 1985 Jamaludeen et al.] 2022 Wu et al.. 2023a occurs where variables at each time step are not independent given their historical observa- tions. Swanson Granger (197)) first discuss the instantaneous dependence from the perspective of noise. Lots of works investigate instantancous dependency from the causal view. Specifically, Hyvärinen et al. (2010) estimate both instantaneous and lagged effects via non-Gaussian model. Gong al. (2015) (017) discover causal structure with instantaneous dependency from subsampled and aggregated data. Recently, Zhu et al. (2023) consider instantaneous dependency in reinforcement learning.
A.5 TIME SERIES FORECASTING
Time series forecasting (ynndman & Athanasopoulos 2018 Box & Pierce 1970) is one of the most popular research problems in recent years. Recently, the deep-learning
methods have made great progress, which can be categorized according to different types of neural architectures like the RNN-based methods (Hochreiter & Schmidhuber 1997 Lai et al. 2018 Salinas et al. (020), CNN-based models (Bai et al.
2018 Wang et al. 2022] Wu et al. 2022p, and the methods based
2021b a)). Recently, Transformer-basee methods (Zhou et al., 2021 Wu et al., [021] Nie (002)) further push the development of time series forecasting. However, these methods seldom consider the instantaneous dependencies of time series data.
on state-space model (Gu et al., 2022
B PROOF OF THEORY
B. RELATIONSIIPS BETWEEN GROUND-TRUTI AND ESTIMATED LATENT VARIABLES
Theorem A1. (Identifiability of Temporally Latent Process) For series of observations x; Rn and estimated latent variables 2; TR with the corresponding process ft, ( where invertible, suppose the process subject to observational equivalence x; = g(2)) Let Ct & (z.-1.2) 21} E R2n and Mc, be the variable set of two consecutive timestamps and the corresponding Markov network respectively. Suppose the following assumptions hold:
I (Smooth and Positive Density): The conditional probability function of the latent variables e, is smooth and positive, ie., p(ce (2-2) is third-order differentiable and (c))))222) > over R22n
A2 (Sufficient Variability): Denote MMel as the number of edges in Markov network Me.- Let
w(m)= = -000 Oct,,02t-2.0
8 log log.(cc./m-2))
8 log (og((cc()s22))
dct 020-2.m
(AI)
8 log p(c//zi-2)
8 log p(c, (2-22)
8 plcp (212))
+
00
de 02: 2.m
0es (US)EE(ME.)
where @ denotes concatenation operation and (i,j) E E(Me.) denotes all pairwise indice such that Ct.s, Ct.j are adjacent in MM..- For m E [1....,,, there exist 4n + |Me different values of
2 such that the 4n + |M..l values of vector functions w(m) are linearly independent.
Then for any two different entries ir,s, &t, of C E R22n that are not adjacent the Markov network Me, over estimated č,
(i) Each ground-trui latent variable Ct of E 122n is a function of at most one of & and én (ii) For each pair of ground-truth latent variables C, and Ct,j of c, E Rmn that are adjacent in Mes
over ct, they can not be function of e and e, respectively.
21

---

Published as a conference paper at ICLR 2025
Proof. We start from the matched marginal distribution develop the relationship between z, and ž, as follows:
p(x,) = p(xx) = p(g( p
p((g = p(z:) = p(h (2.)) = p(z)), (A2)
where g: 2 X' denotes the estimated mixing function, and h
:= :h:=0a g- o g is the transformation between the ground-truth latent variables and the estimated ones. Since 9 and g are invertible, is invertible as well. Since Equation (A2) holds true for all time steps, there must exist an invertible function be such that p(he(er)) - p(ct), whose Jacobian matrix at time step
Jha 0
J Jhat
(A3)
Jhnst
Then for each value of X the Jacobian matrix of the mapping from (x:-2, ē,) to (xx-2, ci) can be written as follows:
I
0 Jho.t
where * denotes any matrix. Since X;-2 can be fully characterized by itself, the left top and right top block are 1 and respectively, and the determinant of this Jacobian matrix is the same hh Therefore, we have:
p(e,, x1-2 p(c,, 2
(A4)
Dividing both sides of Equation
A4) by 2(x)-2), we further have:
p(c,(x(-2) = p(cr|x:-2)]Jnn.d
(A5)
(2)))2222) we have: Since (x-))xx-2)==) = p(c:lg(z(-2)) = p(c, 2)), and similarly (())) (00t))))p@((2_--_
2:2 log (c.2)) + log Jnnizl
(A6)
Let Cr Ct,1 be two different variables that are not adjacent in the estimated Markov network Ma, over C, = (21-1, 2.). conduct the first-order derivative w.r.t. Gτ, and have
i logp(ce((1-2) Hd,
& Ologp(calze-> der. 0e,,
Olog [Jneel
Σ
(A7)
Oe,,
80g
We further conduct the second-order derivative w.r.t. dr and G,,, then we have:
8 Dogp(Cc/zee
(2.2)
82 logp(col(e-2)
der, deg. Thg Jd,,1
06.1
de,
(A8)
2n Olog poogp(cslzz-- (4 Pe.s 2o J 06, +
dc,
Since G,e,C not adjacent in Ma, âg and द., conditionally independent given & (0r,, C.)). Utilizing the fact that conditional independence can lead to zero cross derivative (Lin 1997), for each value of 2,-2, have
82 logg((;/21-2) Je.,
8a loggper (fr E,,3, 22-2) +
82 log p(ér, Melee///rrr ( C 22-2)
00,,
(A9)
82 logg((e (er,, 22) 0.
22

---

Published as a conference paper at ICLR 2025
Bring in Equation A9
Equation
A8
be further derived
0² log pl C2 2
der dēg
dea.
2r
8 &logp(ca((12) der dei. 0ōg De,
0
Oct,
00g,
dcr,
1=1 (-U,)))((Me)))
(i)=)
(ii)ce, ce.s adjacent Mec
8 log (og
dca,
det.
E
dess
Dūs, 00e,
((G.))E(ME.)
(i and on. are not adjacent in Mec
2n log p(ce((=-2)
Pe.s +
θ log Jnnne
+
dc,,
06,
2m1
(A10)
where (j,t) ((Mc.) denotes that Cr and orj are adjacent in Me. Similar Equation A9 when Ct Ct not adjacent in MMe. Thus, Equation A10 be rewritten
have pp(celar-2) as
2n
82 log p(ce 22-2)
dea.
dcr, Dêr,
Σ
2n
Σ
02 log plo/(cc
dea. dei,j
0
Oct,
de,,
Ocr,
deg,
1=1
==1 j-(.))EE(MM))
2n llggp(a (1-22
Pa.i
Ölog [Jnec] 0i,,
di, D0,,
(A11)
Then for each m 1,2, and each value of 2t-2,m- conduct partial derivative both sides of Equation Alll and have:
2n 8 (ogg((c,lz1
dess det.
2n
83 Log de. det.j dc, 021
Σ
dct, 020
06,
0ê,,1
06,, 06,,1
i=1 (..))E((M.)
2n
8 log pl ((c/ 22)
Oct,
Oc,, 021-
06
(A12)
Finally we have
2n 8 log p(c, 21-2)
das, der.
2n 82 loggple (2-2)
Ocq,
dez a2--2m 2.m
06, Fde,
02, 2,m
(A13)
83 loog logp(c;(z(-2)
dea,
deri dêg,
ders dcr.
Σ
+
De,, 0 (..))))E(M.)
2,1m 2,m
0i,,
According to Assumption A2, we can construct 4n + [Mel different equations with different values of m and the coefficients of the equation system they form are linearly independent. To ensure that the right-hand hand side of the equations are always O, the only solution
Oer. deti =0,
(A14)
0e,,
0o,,
Oa, + dey.
Ic,, =0. 0ēa.
(A15)
06g
06,,1
De,t
det, =0.
(A16)
06,
23

---

Published as a conference paper at ICLR 2025
Bringing Eq A14 into Eq A13 That is,
least product must be zero, thus the other must be well.
ac,.j =0. d6, k 06g
da,
(A17)
According the aforementioned results, for any two different entries čτ Ōt, E é, that are not adjacent in the Markov network Ma, over estimated č₁ we draw the following conclusions.
(i) Equation (A14) implies that, each ground-truth latent variable Ct E Ct a function of at most one of Ct,k and Ct.l-
(it) Equation (A17) implies that, for each pair ground-truth latent variables cz, and Ct2 that are adjacent in Mo, over C₁2 they can not be function of & and &τ, respectively.
B.2 EXTENSION TO MULTIPLE LAGS AND SEQUENCE LENGTHS
For the sake of simplicity, we consider only one special case with r = and I, = 2 in Theorem A2 Our identifiability theorem be actually extended arbitrary lags and subsequences easily. For any given T, and subsequence which is centered at with previous lo and following hi steps, i.e., C2 = fzi-lo. ........ Zz. ........ Zethit } E (lo+hi+ 1)xn, In this case, the vector function w(i, j. m) in Sufficient Variability Assumption should be modified
w(i,j.m) m) - 0 log plc,
21-10-1, ,0
to-1o-
8 log p(c, 11-111-1," -1--1--7 1o-7
+
de?, 02c-1o-11m
de, dee,2n 02t-1o-11m
8 log plce
21-10-11111111 1,
Bt
8 log p(ee to-
to- Oct, 024
Zt 10
(A18)
00, -0--
(8' (8'logp/cs (logp(cc//tt- 10--0-11---11 :21-10-7)
ac, 021-
(13))E((Me))
Besides, 2 x n X (lo + hi + 1) + |Me,l values of linearly independent vector functions in z lo 7] and m E [I...., .n] are required as well. The rest part of the
for t' E [t 12-44 lo - 1,.... t theorem remains the same, and the proof can be easily extended in such setting.
B.3 IDENTIFIABLITY OF LATENT VARIABLES
Theorem A2. (Component-wise Identification of Latent Variables with instantaneous dependen- cies.) Suppose that the observations are generated by Equation (7-(2), and Me. is the Markov network over = [z(-1, Zt, Z(+1} E R33". Except for the assumptions Al and A2 from Theorem we further make the following assumption:
A3 (Sparse Latent Process): For any 2 E 2z, the intimate neighbor set of z4 is empty set.
When the observational equivalence is achieved with the minimal number of edges of estimated Markov network of Me,, then have the following two statements:
(i) The estimated Markov network Me, is isomorphic to the ground-truth Markov network Me.-
(ii) There exists permutation of the estimated latent variables, such that 24, and 21,((s) is one-to-one corresponding, iee., ž₁,i is component-win identifiable.
Proof. First, we demonstrate that there always exists a row permutation for each invertible matrix such that the permuted diagonal entries are non-zert (Zhang et al. 2024).. By contradiction, if the product of the diagonal entry of an invertible matrix zero for every row permutation, then we have Equation
det(A) - Σ (gg((a) II (o()))))
(A19)
ocS.
by the Leibniz formula, where S, is the of n-permutations. Thus, we have
Il oo( = O, Yo E Sn,
(A20)
24

---

Published as a conference paper at ICLR 2025
which indicates that det(A) 0 and A is non-invertible. It contradicts the assumption that A is invertible, and a row permutation where the permuted diagonal entries are non-zero must exist. Since h2 is invertible, for z, at time step there exists permuted version of the estimated latent variables, such that
Ōzi + O, i= 1 n, (
(A21)
where is the corresponding permutation at time step t. Since = (2--1, 21, Z by applying Tt-1, Te, Tt+1, we have n' such that
der. (0))
# O, 1, .3..
(A22)
Second, we demonstrate that Me, is identical Mar's where May' denotes the Markov network of the permuted version of T (ê,).
On the hand, for any pair of (i,j) such that Ct. Ct, adjacent in Mc. while (()) (t, are not adjacent in Mar according to Equation (A17). we have 0c,, Oc.,s = 0, which is (6, d',,,'(5)
contradiction with how T' is constructed. Thus, any edge presents in Mc. must exist in Mezz On
the other hand, since observational equivalence can be achieved by the true latent process (g. f. Po..). the true latent process clearly the solution with minimal edges.
Under the sparsity constraint the edges of MMry' the permuted estimated Markov network Mar must be identical the true Markov network Me. Thus, claim that
(i) the estimated Markov network Me, is isomorphic to the ground-truth Markov network Me.
Sequentially, under the same permutation mes we further give the proof that žτ, only the function of ( Since the permutation happens on each time step respectively, the cross-time disentanglement prevented clearly.
Now let us focus on instantancous disentanglement. Suppose there exists pair of indices i,j E {1, .1. m). According to Equation (A21) we have dzs.i 0 and 0s., 0. Let discuss it case by case.
(E.,)(())
If 2₁,a is not adjacent Zt,j- have (,) is not adjacent to ( well according to the conclusion of identical Markov network. Using Equation (A14), we have B3g. Ss,, 83
0, which leads (h,,
0sc.i
If ž is adjacent to Eegi we have (1,x.()) adjacent to (f) When the Assumption A3 (Sparse Latent Process) is assured, i.e., the intimate neighbor set of æt, is empty, there exists at least one pair of (f', such that 2 is adjacent to ž,, but not adjacent to 2t,j- Similarly, we have the same structure the estimated Markov network, which means that Ev a. (k) is adjacent to ((,)((() but not adjacent to (( Using Equation (A17) we have 0=.. 0.
0b.r,(1) ( O. which leads to Bhe,)(G)
8ack
83 = Thus, we have reached the conclusion that In conclusion, we always have 8h.
(ii) there exists permutation of the estimated latent variables, such that 21,5 and &1,((() is one-to-one corresponding, i..., 24,4 is component wise identifiable.
B.4 GENERAL CASE FOR COMPONENT IDENTIFICATIONS
In this part, we briefly give the proof for more general case of our theorem.
Corollary A1. (General Case for Component-wise Identification.) Suppose that the observations
generated by Equation (1-2)) and there exists C₁ (zt-a," " 2+to] TR. [[a+b+1]xx
25

---

Published as a conference paper at ICLR 2025
2r11
1,+1,1
2r-1,1
Zr,1
Zt+11
2t-12 1.2
Zt2
1,+
Z--
2,31.3
2,+
Zt-
1,3
Zr,
1,+1.3
(a) Directed Acyclic Graph
(b) Markov Network
Figure A5: An example of DAG (a) and Markov Network (b).
with the corresponding Markov network Me.. Suppose assumptions Al and A2 hold true, and for any Zt,i E Zt, the intimate neighbor set of 2 is an empty set. When the observational equivalence is achieved with the minimal number of edges of estimated Markov network of Ma. there must exist permutation of the estimated latent variables, such that and ,,((( is one-to-one corresponding. i.e, 2z, is component wise identifiable.
The only difference is that given different subsequence, the variables that are used make intimate neighbors empty might be different. The rest part of the theorem remains the same.
Proof. The proof is similar to that of Theorem A2
Here we further discuss the idea behind the Sparse Latent Process. For latent variables 24,1: 2t,f that are entangled at some certain timestamp, the contextual information can be utilized to recover these variables. Intuitively speaking, when 24,; is directly affected by some previous variable, says 2r-1,k, while Zt.j is not. In this case, the changes that happen 2,-1,k can be captured, which helps to tell 2 from zt.j. Similarly, if žt,; directly affects 114 while žt.jj does not, we can distinguish t,, from æ.j as well. When all variables are naturally conditionally independent, no contextual information will be needed. One more thing to note is that, even though the sparse latent process not fully satisfied, long some structures mentioned above exist, the corresponding entanglement can be prevented.
B.5 IDENTIFIABILITY OF LATENT CAUSAL PROCESS
Building on the results of Theorem 2 the latent variables are component-wis identifiable, which directly implies the identifiability of the latent causal process up to the Markov equivalence class. Leveraging temporal information allows one to further refine this identification beyond the equivalence class. each latent variable 2, has at least one temporal ancestor, one can establish full identifiability of the graph over the latent processes.
Theorem A3. (Identification of Latent Causal Process.) Suppose that the observations are generated by Equation 2) and that Me, is the Markov network over C, {2r-1, 1,,, 2z} & R Suppose that all assumptions for axeorem@holee of adjacent latent variables 2,,, 2 at time step t, their time-delayed parents are not identical, i.e Pua(21,3) * Pas(21.)) Then the causal graph of the latent causal process identifiable.
We further make the following assumption: for any pair
Proof. Since all assumptions for Theorem hold, latent variable z; component-wise identifiable, i.e. there exists a permutation π and invertible functions h4 such that =,, = (())), )- As c2 is nothing but a concatenation of z; from different time steps, C₁ is also component-we identifiable with corresponding n and ht. Therefore, their conditional independence relationships are consistent:
Cri T a.j {a.k & S} + (,,0(() 1 &t,'() (c,,=((K) & S}
(A23)
for all S C 21, 2m j. In this way, transform the problem of causal discovery for latent variables into a problem of causal discovery for observable variables.
We first prove that the skeleton of the estimated causal graph is identical the true skeleton.
26

---

Published as a conference paper at ICLR 2025
1--11
8r,1
F--12
---33
Figure A6:
The example of dataset A whose causal graph is identifiable.
On the one hand, if Ca. and Ct.j are not adjacent, there exists d-separation set Sa such that Cr,i Ct. {ce. | k e Sa)- Mcanwhile, we have (,, (,,)"()) (4, & E Sa}. Thus, (1) and (1,x1() are not adjacent as well.
On the other hand, if C1 and Crg adjacent, there will be d-separation for (1,7'()) (t)) meaning that (() and (t,='()) are adjacent in this case.
In conclusion, the skeleton is identifiable.
Then we prove that the time-delayed edges can identified. Consider any Ct,, from 24-1 and or. from 2z. Similarly, we have (s) from 2;-1 and (x) from 2, Since the skeleton is identified and the direction implied by temporal information, specifically going from time step - 1 to t, there exists an edge from (1)) (t) if and only if cr, points to Ct.j. Thus the temporally latent causal process can be identified.
Finally, determine the direction of instantaneous edges. Since the skeleton is already identifiable, we only need to determine the direction. Consider any pair of Ct2 and Ct, from 2,, where Ce + Ct.j- Since (a,(cr,)) * Paa(Ct.). there exists least Cr from 2 such that Cr points to exactly one of Ct or Ct.j- We proceed with a case-by-case analysis.
In case 1, we have Ct -> Ct,i -> Ct.j where Cr Ct.j are not adjacent, and Cr, is in the d-seperation of Ct and Cr.j. Meanwhile, in the estimated skeleton, (t,)'(a) is also in the d-seperation of (k,z((k) and (., Since the direction of time-delayed edge &,,"(()) + (x,r(()) is the known in the estimated skeleton (,x((K) (,,"(()) C,,+"()) we deduce (,,a'() + (,, based on the d-seperation.
In case 2, we have C, + Ctjj Cr. where Ct,,, Ct, are not adjacent. Similarly, have that (1,)(1)) is not the d-seperation of (,)) and (()) and can be shown that (s) > (t))
In conclusion, the identifiability of the instantancous latent causal graph is established. Consequently, the entire latent causal process, encompassing both the time-delayed and instantaneous components, is identifiable up to the causal graph.
Discussion: To further illustrate this Corollary, we use dataset A as an example, whose causal graph is shown in Figure A6 Since the skeleton and directions of time-delayed edges are straightforward to determine, we primarily focus on analyzing the directions of instantaneous edges within 2,. Since 21-1,3 + 2,,3 21,2 v-structure, 21,3 21,2 can be determined. Since 24-1,1 1 21,1 - 21,2 is a chain, 1,1 and 2,, are not adjacent, and 2,-1,1 >> 2t,1 is known, we have _,, + 2,,2- Thus, the causal graph identifiable. As shown in Figure the model can learn the true causal graph.
B.6 MARKOV NETWORK
Markov network (or Markov random field) is a graphical model that represents the joint distribution of set of random variables using undirected graph.
27

---

Published as a conference paper at ICLR 2025
Definition 3 (Markov Network). Markov network is an undirected graph G (V, with of random variables Xeev, where any two non non-adjacent variables are conditionally independent given all other variables. That is,
Xa T Maad) V(a,b) ( E.
(A24)
Markov Networks and Directed Acyelic Graphs (DAGs) are both graphical models employed to represent joint distributions and to illustrate conditional independence properties. As shown in Figure A5 both of them are utilized to describe the latent causal process, yet they do not have to be equivalent.
B.7 ISOMORPHISM OF MARKOV NETWORKS
Definition 4 (Isomorphism of Markov networks). We let the V--) be the vertical set of any graphs, an isomorphism of Markov networks M and M is bijection between the vertex sets of M and M
f V:V(M) v(ūr)
such that any two vertices u and of M are adjacent in G if and only if f() and f(u) are adjacent
in M.
B.8 ILLUSTRATION OF INTIMATE NEIGHBOR SET
Here, we provide an example for a better understanding of the Intimate Neighbor Set. Take Fig- ure A5]b) as an example. we consider only one time step, says, Zt. { 2r,2 2t.3 forms a clique. Thus we have (2,1) {2,2,, ),,3), since 24, is adjacent to 21,1 and all other neighbours of 24,1, i.e. 21,3- Similarily, we have (((11) { 21,3), (=,1) = {2 2,,2 as well. In this case, none of them is identifiable. In contrast, if we take all 3 time steps into consideration, the Intimate set of all latent varibles becomes empty. For example, 2r,2 is adjacent to 2t,1 but not adjacent to at least one neighbour of It,1, i.e., 1,1, thus =x,2 & V (z,1). Similarly, do this test for all other pairs variables, and the conclusion can be achieved.
C DISCUSSION OF ASSUMPTIONS
To enhance understanding of our theoretical results, we provide some explanations of the assumptions, their connections real-worle scenarios, well the potential boundary of theoretical results.
First, The smooth and positive density assumption standard in the literature on nonlinear ICA (Khemakhem et al. 2020a Yao et al. 2022 Kong et al. 2022), meaning that the latent variables z change continuously based on historical information. For instance, a weather dataset, temperature varies smoothly over time. However, this assumption may be violated if we cannot fully capture the transition probabilities from the observations. To mitigate this, we can sample larger dataset to better estimate the latent causal process.
Second, the sufficient variability assumption has also been commonly adopted for the existing identifiable results temporally causal representation learning (Yao et al. 2022 2022 Chen et al. 2024a). This assumption describes the changeability of latent variables. Take human motion forecasting an example, the latent variable may represent joint location different time steps. The linear independence of the latent variables means that the changes of each joint cannot be linearly represented by others. Besides, while the sufficiency assumption is foundational to the theory of identifiability. it is not excessively restrictive. Even in cases where this assumption is not fully satisfied, is still possible achieve a degree of subspace identifiability (Kong et al. 2022).
Finally, the sparse latent process is the key assumption of our method, which is common in real-world scenarios. Intuitively, in human motion forecasting, the joints of humans can be considered as latent variables. Since there are few connections among the joints of the human body, the latent process described by the skeleton motion trajectory is sparse. Even though the sparsity assumption is not completely satisfied, we can still attain subspace level of identifiability (Li et al. (024).. In this case, each true variable can be a function of, at most, an estimated version of its corresponding variable and those within the intimate set. Let provide simple example here. In video of moving car, it
28

---

Published as a conference paper at ICLR 2025
Abnormal Motion
Normal Motion
Knee
Ankle
Latent Variables
Latent Variables
Observed Variables
Observed Variables
(a) Motion Prediction without
(b) Motion Prediction with Instantancous Dependencies
Instantaneous Dependencies
Figure A7: An example of human motion forecasting, where joints can be considered as latent variables and the latent skeleton decides the motion.
might be hard to have individual identifiability of the separate car wheels and body; however, they can be considered as essential parts of the macro variable 'car'. This macro representation might be sufficient for the purpose of modeling the interactions between the car and other objects.
D THE IMPORTANCE OF CONSIDERING INSTANTANEOUS DEPENDENCE
Here, we provide an example as shown in Figure A7 to show why instantaneous dependence is important in time-series modeling. As shown in Figure (a), without considering the instantaneous dependencies of knee and an ankle, abnormal motion might be generated, where the leg is bent at a distorted angle. Meanwhile, by taking the instantaneous dependencies into account as shown in Figure A7 (b), the predicted motion complies with human physiological structure.
E IMPLEMENTATION DETAILS
E.1 PRIOR LIKELIHOOD DERIVATION
first consider the prior In (Z):)). We start with an illustrative example of stationary latent causal processes with two time-delay latent variables, i.e. [=r,1, At,2 with maximum time lag 1, i.., 2t,, = fz(z(11, Ex,) with mutually independent noises. Then we write this latent process transformation map (note that overload the notation f for transition functions and for the transformation map):
24-1,1
2t-1,1
2t-1,2
21-1,2
At,1
6t,
2r,2
By applying the change variables formula the map f, we can evaluate the joint distribution of the latent variables pl D(21-1,1- 1,1, 21-1, 21,1, 21,2) as
pl 2 1.1. Z 1,2, 2, 20,2) p 21 12, 64,1, 62,2)
(A25)
det Jrl
where Jr is the Jacobian matrix of the map f, where the instantaneous dependencies are assumed to be a low-triangular matrix:
0
Jr
@st.1
Dzy 1.1
Be..1
0
0.y-1,1
Day-1,2
Dec.
29

---

Published as a conference paper at ICLR 2025
Given that this Jacobian is triangular, efficiently compute its determinant thermore, because the noise terms are mutually independent, and hence Ct Ct.j for j o and 1. 7z-1, so we can with the RHS of Equation (A25) as follows
II, Sac, Fur- ci
p(2 1,2, 21,1, 21,2) = p( 1111-111, 1.) x p(eL,, 61,2) = 11a-111 21 x I(,(()))
(A26)
[Je]
[Jr]
Finally, we generalize this example and derive the prior likelihood below. Let {ri)i=1,2.3,.. be set of learned inverse transition functions that take the estimated latent causal variables, and output the noise terms, i.e., é. rs ( {2.-))) Then we design transformation Jacobian as follows:
B with low-triangular
In,XL
21 ... 21 2d]
mapped to ".-,," ... 2 1. c,,a]
with JA-BB
diag
Or O
(A27)
Similar to Equation A26 as:
obtain the joint distribution of the estimated dynamics subspace
the
logp(A) log p (2 L. 21-1) +
Σ
log p(e,,) log[[ee((AABB)))
(A28)
Because of mutually independent noise assumption
Finally, we have:
log ((2:((21-20 brmm) = Σ p(éc,) +
Σ log
0r:
(A29)
0
Since the prior of p(2 (1:r) assumption, we can estimate p(ž1+1:1 (2::t) similar way.
Ti==+11 p(2.) (2:-1) with the assumption of first-order Markov
E.2 EVIDENT LOWER BOUND
In this subsection, we show the evident lower bound. We first factorize the conditional distribution according to the Bayes theorem.
In Inx(xi:) In p(x1.T 21:T)
p(X1T,, Z1TT (
p(21-T X1:T)
p(z1.1 X1:1 (Z1rT/11
(A30)
2 Eg(zur x.r) In p( (Z::T) Dr (g(z)T (2. x x (Pp()T) ELBO
LELD
As for the time-series forecasting task, we let X1:: and X++1:7 be the historical and future observed variables, then the ELBO can be further derived as follows:
In 1( x1a) In P 21.r, X1:e) In p(x++1.7 Z1:1, Ze+1T X1:))
p(z1.T X(+1T)
> (pprrii In p(x(+1:1 Z4+ Z1.))p
(Z x1:))
=Egtarledd In p(x ZI:r)
(a In pnp(x+1111b (2.:))
DxL(g(Zz.1 (1:)))((Zz:)))
Epme
CRLD
(A31)
where L, denotes the reconstruct loss of historical data, Cpre denotes the forecasting loss of future data.
E.3 MODEL DETAILS
We choose MICN Wang et al. (2022) the encoder backbone of our model on real-worle datasets. Specifically, given the MICN extracts the hidden feature, we apply a variational inference block and then MI. based decoder. Architecture details of the proposed method shown in Table A4
30

---

Published as a conference paper at ICLR 2025
Table A4: Architecture details. 7, length of time series. xx| input dimension. n: latent dimension.
LeakyReLU: Leaky Rectified Linear Unit. Tanh: Hyperbolic tangent function.
Configuration
Description
Output
@
Latent Variable Encoder
Input:xi:
Observed time series
Batch Sizextx dimension
Dense
Ix, neurons concalenation
Batch Sixextxlxal
Concal zero
Batch SizexTxlxa
Dense
n neurons
Batch Size xT'n
M
Decoder
Input:Zi:T
Latent Variable lx, neurons, Tanh
Batch SizexTxn
Dense
Batch Size
Modular Prior Networks
Input: 71.T
Latent Variable
Batch (n+1) (n+1)x122
Dense
128 neurons,LeeakyR RU
Dense
128 neurons,Leaak ReLU
128x 128
Dense
128 neurons,LeakyReL..
128x 128
Dense
neuron
Batch Size
Jacobian Compute
Compute log(det(J))
Batch Size
E.3.1 REPRODUCIBILITY OF SIMULATION EXPERIMENTS.
For the implementation of baseline models, we utilized publicly released code for TDRL and iCRITIS. However, since the author did not release the code for G-CaRL, we implemented ourselves based on the paper. It is important to note that the original code of iCRITIS only accepts images as input. To adapt it to our needs, we replaced the encoder and decoder with a Variational Autoencoder, with the same hyperparameters used in IDOL. Our code is modified based on the code of TDRL, and shared hyperparameters remain the same.
When calculating the MCC for all methods, we use the mean value from the VAE encoder. Regarding the latent causal graph, we employ different approaches depending on the method. For IDOL, TDRL, and G-CaRL, utilize the Jacobian matrix proxy for capturing the causal process. On the other hand, for iCRITIS, we rely on the "get_adi_matri((" function provided in its original code to obtain the latent causal graph.
E.3.2 REPRODUCIBILITY OF REAL-WORLD EXPERIMENTS.
The model details of our method are shown in Table A4 For fair comparison, employ the official implementations and use the default hyperparameters. Since our model is modified from the model and the baselines share the same hyperparameters.
official implementations of TDRL, both To achieve the results from the well-fit baselines, we employed the default hyper-parameters and tried different values of learning rate for the best models. Please refer Table A5 to All for the implementation details of the baselines.
31

---

Published as a conference paper at ICLR 2025
Table A5: TDRL architecture details. T: length of time series, [zy]: input dimension. n: latent
dimension. LeakyReLU: Leaky Rectified Linear Unit. Tanh: Hyperbolic tangent function. Ouiput
Configuration Description
Latent Variable Encoder
Input:
Observed time series
Batch Size xtx X dimension
Dense
[ra]] neurons concatenation
Batch Size xtx [zy]
Concat zero
Batch Size xTx [re
Dense
n neurons
Batch Size x Tn
Decoder
Inputzl::
Latent Variable
Batch Size x'nn
Dense
neurons, Tanh
Batch Size x x zt
Modular Prior Networks
Inputzl::
Latent Variable
Batch Sizex(n+1) (n+1)x 128
Dense
128 Leaky
Dense
128 Leurons,,eakyyReL. UeLU
128x 128
Dense
128 Leakons,
128x 128
Dense
I neuron
Batch Size
JacobianCompute
Compute log(det(J)
Batch Size
Table A6: CARD architecture details. T: length of time series. |æy]: input dimension. patch
number. n: patch length.
Configuration
Description
Output
Input:Zi:t
Observed time series
Batch Sizextx X dimension
Permute
Matrix Transpose unfold add d neurons, dropout
BSx xzy X
unfold
BS x Øt X pxn
Add random
BSX I4 Xpxп
Dense
BSx [7g xpxd
Concat random
concatenation
BS xt, X (p+1 x
Dense
Attenion reshape
BS x x3 X (p+l)xd
reshape
BS x Ø7 X (p+1))xd)
Dense
(T-t) neurons
BS x ax X (T-1)
Permute
Matrix Transpose
BS Zt
Table A7: iTransformer architecture details. T: length of time series. |a,]: input dimension.
Configuration
Description
Output
Input:21::
Observed time series
Batch Sixe X dimension
Permute
Matrix Transpose
BSx 2x xt
Dense
Embedding
BSx .t: x
Dense
Attention
BS X
I3 x
Dense
T-t neurons
BS It X (T-()
Permute
Matrix Transpose
BS (T-t)
32

---

Published as a conference paper at ICLR 2025
Table A8: Auoformer architecture details. T: length of time series. [xa] input dimension.
Configuration
Description
Output
Input: seasonal, trend seasonal concat zero trend concat x mean seasonal
Observed time series
Batch Sizextx X dimension
AvgPoolld concatenation concatenation Embedding
BSxtx [xe
BSxTx I
BSxTx I+ BSxTxd
Input:rr::
Observed time series
Batch Sizextx X dimension
Dense
Embedding Attention
BSxtxd BSxtxd
__enc
Inputxx_ enc, seasonal
_nn
Conv ld
XSxTxd BSxTx aT
Dense
Convld
Add trend
Add
BSxTx
Table A9: TimesNet architecture details. T: length of time series. [zy]: input dimension.
Configuration
Description
Output
In
Observed time series
Batch Sizextx X dimension
Dense
Embedding
BSxtxd
Permute
Matrix Transpose
BSxdxt
Dense
T neurons
BSxdxT
Permute
Matrix Transpose
BSxTxd
Dense
Convld.LayerNont
BSxTxd
Dense
x3 neurons
BSxTx[n [
Table A10: MICN architecture details. T: length of time series. |æy]: input dimension. LeakyReLU:
Leaky Rectified Linear Unit. Tanh: Hyperbolic tangent function. Description
Configuration
Output
Input:0i:< scasonal, trend seasonal concat zero
Observed time series
Batch Sizextx X dimension
AvgPoolld
BSxtx [ca]
concatenation
BSxTx |
trend permute
Matrix Transpose
BSx [2y xt
trend
(T-t) neurons
BS (Za (T-t)
trend permute
Matrix Transpose
(S- [as]]
scasonal
neurons
BSxTxd
scasonal
Conv LayeryyrNorm, Tanh
BSxTxd
seasonal
[x neurons
BSxTx |
scasonal add trend
Add
BSx (T-() 24
Table A11: FTTS architecture details. length of time series. |az] input dimension. p: patch number. patch length.
Configuration
Description
Output
Input:71:t
Observed time series
Batch Sizextx X dimension
FFT rfft
(122+11+ [
Permute
Matrix Transpose
BS x X (1/++1)
Dense
T/2+1 neurons
BS .r X (T/2+1)
Permute
Matrix Transpose
BS (T/2+1)x
IFFT
irfft
BS BSxTx [xy]
33

---

Published as a conference paper at ICLR 2025
Table A12: Standard Deviation results simulation data.
Datasets
IDOL
TDRL
G-CaRL
iCITRIS 3 T-VAE
SlowVAE
iVAE
FactorVAE
PCL.
TCL.
A
0.029
0.045
0.003
0.105
0.014
0.025
0.064
0.025
0.011
0.008
B
0.013
0.022
0.004
0.039
0.013
0.005
0.011
0.035
0.073
0.028
CCCC
0.002
0.028
0.003
0.039
0.031
0.010
0.024
0.027
0.021
0.038
D
0.006
0.003
0.051
0.010
0.028
0.074
0.034
0.027
0.033
0.007
0.037
0.005
0.001
0.021
0.015
0.002
0.009
0.002
0.039
0.016
F
0.029
0.018
0.004
0.165
0.134
0.014
0.031
0.018
0.007
Table A13: MCC results of IDOL and IDOL-S with mean and standard deviation.
IDOL
IDOL-S
MCC
0.8595 (0.0599)
0.8498 (0.0481)
F
EXPERIMENT DETAILS
F.1 SIMULATION EXPERIMENT
F.1.1 DATA GENERATION PROCESS
As for the temporally latent processes, we use MLPs with the activation function of LeakyReLU to model the sparse time-delayed and instantaneous relationships of temporally latent variables. For all datasets, we set sequence length as 5 and transition lag as That is:
2,, (Leaky (W,; 1 0.2) + VeLs Zt,<s) CtSs 61,s
where W.,; is the i-th row of W and Ver.s is the first columns in the i-th row of V. Moreover, each independent noise et,; is sampled from the distribution of normal distribution. We further let the data generation process from latent variables to observed variables be MLLs with the LeakyReLUU units.
We provide 6 synthetic datasets, with 3,5,8,888.16 16 latent variables from A F. For dataset A, have Wa = [[1, 1. 0J. (0, 1, 0J, [0.[000 0, 1]]. For dataset B, we have WB as an eye matrix with 2 extra nonzero entries. For dataset C, F, we have Wo, Wr as eye matrices. For datasets D and Wp and WE are dense matrices with all nonzero entries, which are generated by the data generator of TDRL Yao al. 2022). When it comes to we have and only have V = 1 Vi for dataset A, B, C, F, F. We also set VD O, which means that there are no instantaneous effects.
The total size of the dataset is 100,000, with 1,024 samples designated as the validation set. The remaining samples are the training set.
F.1.2 EVALUATION METRICS.
To evaluate the identifiability performance of our method under instantaneous dependencies, employ the Mean Correlation Coefficient (MCC) between the ground-truth z, and the estimated &. higher MCC denotes better identification performance the model can achieve. In addition. also draw the estimated latent causal process to validate our method. Since the estimated transition function will be a transformation of the ground truth, we do compare their exact values, but only the activated entries.
F.1.3 MORE SIMULATION EXPERIMENT RESULTS
We run each experiment 3 times, with seeds 769, 770, 771. Standard deviation results of the simulation datasets are shown in Table A12
1.1.4 ABLATION STUDY
To further illustrate the significance of sparsity, compare IDOL model with its variant that lacks a sparsity constraint (IDOL-S). The experimental results on the synthetic dataset are presented in Table A13 The result shows that sparsity constraint crucial to the model.
34

---

Published as a conference paper at ICLR 2025
Table A14: MSE and MAE results of different methods the transformed HumanEva- datase.
Prodict Length
IDOL
TDRL
CARD
MICN
Transformer
TimesNet
Autoformer
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSB
MAE
MSE
MAB
MSE
MAB
MSE
MAB
125
0.082
0.196
0.084
0.202
0.127
0.251
0.108
0.237
0.083
0.201
0.105
0.227
0.111
0.235
0.133
0.267
250
0.101
0.226
0.134
0.274
0.201
0.319
0.141
0.273
0.107
0.236
0.161
0.287
0.192
0.312
0.188
0.326
0.244
0.144
0.288
0.230
0.342
0.180
0.306
0.115
0.267
0.193
0.315
0.226
0.341
0.224
0.352
125
0.091
0.220
0.097
0.235
0.098
0.227
0.168
0.305
0.095
0.225
0.097
0.223
0.117
0.247
0.141
0.283
250
0.103
0.254
0.139
0.285
0.144
0.274
0.183
0.320
0.130
0.272
0.123
0.260
0.134
0.271
0.154
0.300
375
0.138
0.276
0.155
0.296
0.179
0.310
0.182
0.321
0.154
0.295
0.146
0.286
0.165
0.308
0.161
0.30X
1.337
0.834
1.531
0.787
2.306
1.027
1.259
0.786
1.488
0.793
1,905
0.911
2.941
1.210
250
1.961
1,045
2.113
1.307
3.121
1.316
3.221
1.293
2/198
1.346
3.152
1,383
3.90S
1.513
1.169
1.633
375
2.326
1.164
2,514
1.326
3,948
1,558
4,020
1.537
2,377
1.274
4.075
1.631
4.315
1.662
4,951
1.815
125
0.36
0.405
0.373
0.418
0.512
0.482
0.595
0,543
0.579
0.425
0.517
0.490
0.523
0.501
0.785
0.632
0.923
0.710
0.569
0.532
0.813
0.655
0815
0.656
1.061
0.769
0.649
0.653
0.581
0.980
0.720
1.075
0.781
0.655
0.587
0.959
0.721
0.966
0.726
1.193
0.826
125
0.045
0.162
0.047
0.164
0.122
0.254
0.288
0.431
0.046
0.162
0.124
0.257
0.067
0.184
0.314
0.405
250
0.104
0.256
0.143
0.302
0.333
0.434
0.589
0.618
0.141
0.502
0.340
0.488
0.170
0.292
0.629
0.614
0.331
0.431
0.508
0.691
0.673
0.175
0.341
0.425
0.253
0.370
1.048
0.761
125
0
0.189
0.078
0:214
D.118
0.262
0.227
0.385
0.071
0.204
0.115
0.256
0.131
0.271
03223
0.361
WT
250
0.127
0.277
0.152
0.311
0:297
0.421
0.407
0.520
0.153
0.312
0.287
0.44
0.195
0.338
0.395
0.509
375
0.151
0.385
0.171
0.328
0.365
0.473
0.425
0.530
0.169
0.327
0.313
0.441
0.289
0.431
0.430
0.538
Table A15: Standard deviation of MSE and MAE results on the different motion.
Predict
IDO1.
TIRI
CARD
FITS
MICN
iTrnsformer
TimrsNer
Ausefeener
Length
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MAE
0.0013
0.0000
0.0100
0,0006
0.0005
0.0022
0.0022
0.0026
0,0029
0.0004
0.005
0,0002
0.0023
0.0011
0.0144
0.0177
0.0115
0.0855
0.0499
0,0253
0.0026
0.004
0.0022
0.0081
0.0256
0.0111
0,0872
0.0296
00238
0.0171
H3MM
0,0099
000611
0.0088
0,083
0.0108
0,0000
0,0023
0.0010
0,006
0.0135
0.0183
0.0116
0.8377
0.0025
0.085
0.0099
0,0003
0.0006
0.0126
0.0123
0.0428
00130
0,0030
0.0000
0.0088
0.0200
0.0099
0,0109
0.005
0,0133
25
0.0001
0.0005
0.0009
0.0225
0,0099
0.0034
0,0009
0.002
0.0001
0.0100
0.000
0.0004
0.0015
0.023
0.026
0.0055
TC
50
0.0044
0.802
0.3011
0.023
0
0.00%
0.0033
0,010
0U0003
0.0133
0.000
0.0044
0.007
0,046
0.821
0.0042
73
0,0015
0.826
0.0027
0.0022
0.0019
0.0030
0,000
0.0022
000004
000055
0.0002
0.0004
0,0029
0.078
00031
0,0066
0.0099
00039
0.0016
0,000
0.0053
0,0066
0.0061
0.009
000005
0.0122
0
0,0055
0.003
04027
00837
0.07%
W
50
0.000
0.0009
0,020
00019
0J9432
0.0136
0.0009
00004
100005
000055
0.0022
0.003
0,0044
02075
B0660
0.0419
75
0.0154
0.017
000011
00019
0L0630
0.0311
0,0033
0.005
100009
000144
0.0035
0.0017
0,0300
0.0069
0.499
B.0349
0.0002
0002
000111
0005
00012
0,000
0,00
00013
000000
0:0017
0,0015
0,015
0,048
0,007
0007
0,00S
0.0012
00015
0
00024
01024
0006
8008
000000
0000
00000
0:0013
0,0099
0,00
06022
0,008
6004
0,0422
0.0056
80062
0
00001
00000
00109
00064
0:0000
0.000
00000
0:0026
0,0033
00001
0.063
0.0012
00043
0.0010
0.003
0.00900
0,0866
00111
0.0026
000000
00055
000000
0:005
0.0055
0.0054
000166
03019
0.0076
0.1744
00105
00147
0,0933
0.009
0,0027
01022
0,0011
0.0063
0.8071
0,000
03018
00003
0
0.0014
0.0017
0,0005
0,005
000099
0,034
0,0013
00004
0,0077
0,0085
0.0079
0,0035
03036
0.0466
0,0043
0.454
0.0465
0,029
0.002
0.3698
0.774
B0000
0.019
00225
0.0001
00015
0L1735
00151
0,011
0.006
0.5525
00708
0,0993
00168
0.089
0,035
0.1308
0.339
0.0051
00330
0,243
125
0,0056
00001
0.019
1,0116
0.0455
0,008
0.0041
0.027
00157
0:0155
0.0131
0,099
1.0019
0.0167
00%2
0,0255
SD
250
0.0044
00002
0.0055
000077
0,44SS
00176
0.0008
00011
0,029
0,0146
0.0064
0,009
0.072
0,069
00051
02017
375
0.0149
0.145
0.0072
0.0079
0.0107
0/044
0,0025
000144
0.0000
0.0200
0.01
0,007
0.0105
080011
00336
0,0000
125
0.0099
0.0945
0,0078
0.010
0.0044
0,001
0.0037
0U009
102016
0.0344
0.0062
0.8387
0,002
03032
0.1585
0.1221
W
250
0.0022
00105
0,226
00003
001157
00127
0,0012
00006
00125
0.0122
0.0130
0.0011
0,0005
00107
0133:
0,7488
375
0.0089
00108
00273
00096
00005
00055
0.0010
00014
00212
0.0035
0.0121
00088
0.008
00006
01389
0,0771
125
0.003
00009
0.065
0,005
000099
0,0866
0,0032
00085
00009
0.0022
0.0021
010027
00042
00455
0,0455
WT
250
0.0044
00000
0.044
0,066
00113
0000
0.0007
00000
00054
00085
0.013
00111
00054
00921
00168
0,0152
375
0,0145
00188
0.0020
0,0027
0.433
0,283
0,0019
0.0044
00002
0,0027
0.0077
00004
0,022
0,0246
00004
0,0043
35

---

Published as a conference paper at ICLR 2025
Table A16: Sample number of different datasets.
Dataset
Size
Gestures
1686
Jog
2050
Humaneva
ThrowCatch
1021
Walking
2454
Discussion
13759
Greeting
8416
Purchases
7480
Human
SittingDown
16438
Walking
16257
Walk Togethr
10658
F.2 REAL-WORLD EXPERIMENTS
F.2.1 REAL-WORLD DATASET DESCRIPTION
Human3.6M Dataset is collected over 3.6 million different human poses, viewed from 4 differ- ent angles, using accurate human motion capture system. The motions were executed by 11 professional actors, and cover diverse set of everyday scenarios including conversations, eating, greeting, talking the phone, posing, sitting, smoking, taking photos, waiting, walking in various non-typical scenarios. We randomly use four motions, i.e., Gestures (Ge), Jog (J), CatchThrov (CT), and Walking (W), for the task of the human motion forecasting. The data is obtained from the joint angles (provided by Vicon's skeleton fiuting procedure) by applying forward kinematics on the skeleton of the subject. The parametrization called relative because there a specially designated joint, usually called the root (roughly corresponding to the pelvis bone position), which is taken as the center of the coordinate system, while the other joints are estimated relative it.
In this dataset, the joint positions can be considered as latent variables. And the kinematic representa- tion can be considered as the observed variables. The kinematic representation (KR) considers the relative joint angles between limbs. We consider the process from joint position to joint angles as mixture process.
HumanEva-1 dataset comprises 3 subjects each performing several action categories. Each pose has 15 joints with three axis. We choose motions, i.e., Discussion (D), Greeting (Gr), Purchases (P), SittingDown (SD), Walking (W), and WalkTogether (WT) for the task of human motion forecasting. Specifically, the ground truth motion the body was captured using a commercial motion capture (MoCap) system from ViconPeak.55 The system uses reflective markers and six IM-pixel cameras to recover the 3D position the markers and thereby estimate the 3D articulated pose the body. We consider the joints latent variables and the signals recorded from the system observations.
The sample number of different datasets are shown in Table A16
F.2.2 MORE EXPERIMENTS RESULTS
We further consider more complex mixture process. To achieve is, we further apply transformation on the observed variables, i.e., X, = fx()), where fo is linear transformation. Then we can consider the sensors as latent variables with instantaneous dependencies and conduct motion forecasting on the transformed datasets. Experiment results the transformed HumanEva-1 dataset are shown in Table A14
F.2.3 QUALITATIVE RESULTS
To further show the effectiveness of the proposed method, we also randomly select a batch of test samples and visualize the forecasting results of different baselines. Visualization results in Figure
A9 show the predicted results of "Discussion". "Greeting", and "Purchases", respectively, which illustrate how the forecasting results align the ground-truth motions, where the red lines denote the ground-truth motions and the lines in other colors denote the forecasting results of different methods. According to the experiment results, find that the forecasting results of our method achieve
36

---

Published as a conference paper at ICLR 2025
Jog(125)-MSE
Jog(175)-MSE
6.3
0.5
623
MSR
MSR
0.3
MAE
2.3-
6:0
300
IDOL IDOL-K IDOL-S
9x
IDOL
IDOL-K IDOL-S
IDOL IDOL-K IDOL-S
(gg(125)-MAE
1og(150)-MAE
[og(175)-MAE
0:31
325
MAE
MAE
012
014
IDOL
IDOL-K IDOL-S
IDOL
IDOL-K
IDOL-S
IDOL
IDOL-K IDOL-S
IDOL
IDOL-K
IDOL-S
Figure A8: Ablation study on the Jog motion. we explore the impact of different loss terms.
IDOL
MICN
iTransformer
TimesNet
TDRL
CARD
FITS
A Auooformer
Figure A9: The illustration of visualization of different methods of Walking, The red lines denote the ground-truth motions, those lines in other colors denote the prediction motions of different methods.
the best alignment of the ground truth. In the meanwhile, the other methods that do not consider the instantaneous dependencies of latent variables may generate some exaggerated body movements, for example, the hyperflexed knee joint and the actions that violate the physics rules, which reflects the importance of considering the instantaneous dependencies of latent causal process of time series data.
F.2.4 ABLATION STUDY
To evaluate the effectiveness of individual loss terms, we also devise the two model variants. 1) IDOL-K: remove the KL divergence restriction of prior estimation. 2) IDOL-S: remove sparsity restriction latent dynamics. Experiment results on the Jog dataset are shown in Figure A8 We can find that the sparsity restriction of latent dynamics plays an important role in the model performance. reflecting that the restriction of latent dynamics can benefit the identifiability of latent variables. We also discover that incorporating the KL divergence has a positive impact the overall performance of the model, which shows the necessity of identifiability.
G COMPLEXITY ANALYSIS
G. WALL-CLOCK TRAINING TIMES
To evaluate the computational complexity of our method, provide the wall-clock training times of different methods. Specifically, we use consistent hardware setup including the same GPU, CPU, and memory configurations for each model ensure comparability. Sequentially, in codes,
37

---

Published as a conference paper at ICLR 2025
Table A17: Mean and standard deviation of wall-clock training times for one epoch.
Meehoos
IIOI.
TDRI.
CARD
PITS
MICN
iTrnsffomer
TimesNet
Autoformer
65.960(7.556)
33.902(5.781)
(2.995(3.035)
92.941(0.715)
45.547(13.034)
38.155(6.441)
324.941(55.286) 61.633)5.673)
MOMOYY PO
MErrICy POREDrIEE
0.005
1G
4.30
0.20
0:05
0.802
20:20
2.09
941
0.011
$24.38
0.99
0.000
0.00
(a) Model Efficiency the Low- dimension Dataset (Human Dataset).
(b) Model Efficiency on the High- dimension Dataset (MoCap Dataset).
Figure A10: Model efficiency comparison for one training step different datasets.
wrap the training process in a timer to measure the actual elapsed time from start finish of one epoch. For each method, we repeat three different times and further report the mean and standard deviation. The wall-clock training times are shown in Table A17
Based on the experimental results, we draw the following conclusions:
Compared to our baseline model TDRL, the wall-clock training time of the proposed IDOL is nearly twice slow. Theoretically speaking, the primary computational difference is that IDOL calculates the Jacobian matrix for both time-delayed and instantancous relationships, while TDRL only calculates the time-delayed component.
Compared to other mainstream baselines, our IDOL method is slower than some models like MICN and iTransformer. However, our method still faster than models like FITS.
G.2 MODEL EFFICIENCY ANALYSIS
To evaluate the model efficiency, we provide model efficiency comparison the low-dimension dataset (e.g., the Human dataset) and the high-dimension dataset (the MoCap dataset) by evaluating the model efficiency from three aspects: forecasting performance, training speed, and memory footprint. As shown in Figure A10 in low-dimensional datasets, IDOL performs nearly as effi- ciently top methods like Autoformer and MICN and outperforms others like CARD. However, in high-dimensional datasets (117 observations), IDOL requires more training time due to the added complexity of Jacobian calculations for instantaneous effects.
H MORE DISCUSSIONS
H.1 ANALYSIS OF
DATA
Identifying causal relationships in high-dimensional latent variable settings a well-recognized challenge the causal inference community (Lopez et al.] [022]] Cheng al.) 2024). As the dimensionality increases, the complexity of identilying causal structures grows due to the expanding search space. We conduct experiments on both synthetic and real-world datasets to verify it and propose a potential solution.
H.1.1 SIMULATION EXPERIMENTS
As for the experiment on the synthetic datasets, we follow the same data generation process to generate simulation datasets with latent variable dimensions of 16, 24, and 32, All these datasets
38

---

Published as a conference paper at ICLR 2025
Table A18:
Experiments results of IDOL simulation data different dimensions of latent
variables.
Dimension
8
16
24
32
MCC
0.9801(0.002)
0.9747 (0.0029)
0.9243 (0.0173)
0.8640 (0.0071)
Table A19: Experiment results of the CMU MoCap datasets.
Mexion
Predicted Longth
IDOL
TDRL
CARD
FITS
MICN
iTranffrrmer
TimesNet
Autuffommrr
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Running
0.988
50-30
0.286
0.109
1,779
0.167
2.217
0.153
0.266
0.103
0.978
0.161
1,831
0.170
4.720
0.252
3.944
0.283
Soceer
50-50
50-25
0.079
0.022
0,043
0,071
0.026
0.084
0.017
0.033
0.206
0.397
0.108
0.082
0.076
0.265
0.065
0,103
0,284
0.057
0.066
0.120
0.133
0.032
0.051
0.082
0.063
0,392
0.068
0.107
0.211
0.452
0.143
0.105
share similar latent causal process: chain-like instantaneous effects and ono-to-one temporal effects. We then measured the Mean Correlation Coefficient (MCC) between the ground truth 2, and the estimated 2, The experimental results are presented in Table AT88 According the experiment result, as the dimension of latent variables increases, the value of MCC is still acceptable, despite possible performance loss in high-dimensional problems.
H.1.2 REAL-WORLD EXPERIMENTS
For the real-world datasets, we consider the CMU-MoCap datase It contains various motion capture recordings and 117 skeleton-based measurements, which is higher dimensionality compared to the Human dataset. We choose the Running and Soccer motions and take the input-prediction length as 50-25 and 50-50, respectively. As shown in Table the IDOL model achieved comparable forecasting performance in the high- high-dimensional dataset Running.
H.1.3 POTENTIAL SOLUTION FOR HIGHI-DIMENSION DATA
To better address this challenge, here we propose several potential solutions to more effectively address the challenges of dibeeniinnii time-series data. One idea is make of the divide- and-conquer strategy. One possible way is to leverage independent relations in the measure time series data, if any. For instance, if processes X; == { |t E T) and X₂ = { T) happen to be independent of processes X3 and X4, then we can just learn the underlying processes for (X1, X2) and (X3, X4)) separately. Another potential way to use the conditional independent relations in the measured time series data. For example, if processes X and X2 are independent from X3 and X4 given X5 and X6, then we can just learn the underlying processes for (X1, X2, Xs. Xx) and (Xs, X4, Xs, X6) separately. In this way, we can reduce the search space and further reduce the complexity even in high-dimensional time series data. We hope that some other developments in reducing the computational load in deep learning can also be helpful.
H.2 NOISY ENVIRONMENTS
In some real-world scenarios, it is possible for the mixing process not to be invertible, for example, if the mixing process is highly noisy in each observed process, which might be the case financial data (Hu & Schennach] 2008). There are some developments relying on the additive noise assumptions. If one makes strong assumptions on the noise, such as assuming an additive noise model, it is possible to develop a certain type of identifiability just like the extension of nonlinear ICA to the additive noise model case in (Khemakhem et al. 2020a). However, general approaches to deal with non-parametric noise terms to developed in this field in the future. To further evaluate the insight of this potential solution, we have conducted experiments on synthetic data. Specifically, we first follow Equation (2) to generate latent variables with temporal and instantancous dependencies and generate observed variables by X, = g(2,, E). where & is the noise introduced to the mixing process. Experiment results in Table A20 show that method can still achieve relatively good identifiability results even in noisy environment, proving the potential of our insight.
hhtp://moccppcssccm...uu
39

---

Published as a conference paper at ICLR 2025
Table A20: The mean and standard deviation of MCC of the IDOL model under noise environment with different noise scales. The noise scale is defined as the ratio of the noise variance to the observation variance, expressed percentage.
Noise Scale
0.1
0.3
0.5
0.7
MCC
0.9257
0.8362
0.7381
0.6095
Table A21:
The mean and standard deviation of MCC on subsets of dataset A with different sizes.
Sample Size
1,000
10,000
25000
50,000
100,000
MCC
0.853(0.102)
0.884(0.011)
0.912(0.025) 0.945(0.023)
0.965(0.029)
H.3 ANALYSIS OF SAMPLE SIZE
To assess the impact of sample size on identification performance, we conduct an ablation study focusing sample size. Specifically, we create subsets of Dataset A containing 1,000, 10,000, 25,000, and 50,000 samples. For each dataset, we used the same hyperparameters, such as learning rate, random seed, and batch size. We repeated the experiment three times with different random seeds and reported the values of mean and standard deviation. The results Table A2T show that as the sample size decreases, the performance of the model gradually declines. However, with just Ik samples, our method achieves relatively good performance, demonstrating its robustness in small sample scenarios.
H.4 SUBSAMPLED TIME SERIES
Sequentially, we consider another case where the time series data are sampled with low resolutions. When the time series data is sampled with low frequency, additional edges are introduced into the Markov network, making it denser. We can further assume sparse mixing process /Zheng] ict al. (2022) to achieve identifiability. Specifically, When conditioned on historical information that provides sufficient changes, the sparse mixing procedure assumption imposes structural constraints on the mapping from estimated to true latent variables. This compensates for potentially insufficient distribution changes, enabling identifiability even when the time series data with low-sample regimes. To further evaluate this insight, we also conduct experiments on synthetic datasets with low-sample regimes. Specifically, we first generate time series data with temporal and instantaneous dependencies. Then we randomly subsample the synthetic time series data. To enforce the sparsity of the estimated mixing process, we add extra L1 constraint regarding the partial differential between the estimated observed and latent variables, i.c. o2. 1. Experiment results are shown in Table A22
Table A22: The mean and standard deviation of MCC between the standard IDOL and the IDOL with sparse mixing constraint.
Model
IDOL.+Sparse Mixing Constraint
IDOL
MCC
0.837(0.078)
0.786(0.085)
From the experimental results, sampled at low frequency, the identifiable performance of the IDOL model decreases. This because the causal process between latent variables becomes dense, which is consistent with theoretical results. 2) by employing a sparse mixing constraint, we found that the identifiability performance of the model has been significantly improved, which proves that the above insight is effective.
draw the following conclusions: 1) When time series data is
40


Reviews:
[{"review_id": "82iJJTJC20", "reviewer": "Reviewer_mQKF", "summary": "This paper proposes IDOL, a framework for achieving identifiability in sequential latent variable models with instantaneous dependencies. The authors establish identifiability up to permutation of the latent variables and demonstrate that the underlying causal graph can be identified up to its Markov equivalence class (if this interpretation is correct). They thoroughly discuss the limitations of their assumptions in comparison to recent works, which helps underscore the significance of the proposed framework.\n\nAn estimation method is also introduced, with experiments on synthetic data verifying the theoretical results, while real-world experiments highlight the importance of incorporating instantaneous dependencies.\n\nEdit: All the major concerns have been addressed in the rebuttal, and hence I have raised my score to 8.", "questions": "- line 41: do you mean “mixing function” instead of “mixture function”?\n- lines 332, 386 and 387: Notation. You are using $\\mathcal{L}$ and $L$ interchangeably. Could you revise this?\n- If I am not mistaken, your identifiability theory does not obtain the causal graph, but a markov equivalence of it (please correct if mistaken). Yet apparently, the synthetic experiments suggest that you estimate the instantaneous causal graph with 100% accuracy (Figure 4, bottom left). Could you provide some explanation for this? For example, is it possible that your assumptions allow for stronger identifiability results that are overlooked in the presented theory?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The manuscript is clear in terms of motivating the problem and introducing the theoretical framework.\n- Incorporation of instantaneous effects into sequential latent variable models is a very significant contribution.\n- The paper discusses limitations of the assumptions in comparison to recent works.\n- The experiments with real-world data motivate the incorporation of instantaneous effects.", "weaknesses": "**Minor Concerns**\n- **Computational Complexity:** The sparsity constraint introduced in Eq. (11) seems to introduce significant computational complexity to the algorithm. The paper would benefit from a more detailed analysis regarding this. For example, would it be possible to compute wall-clock times (in training) for IDOL in comparison the proposed baselines?\n- **Scalability to High-Dimensional Data:** The authors acknowledge limitations with respect to high-dimensional data, which can restrict the application to real-world scenarios. An experiment to understand how high-dimensional one can go with IDOL would be ideal to support your point to understand how high one could go for IDOL.\n\n**Major Concern: Theory Section Clarity and Limitations**\n\nThe paper’s theoretical claims, particularly around identifiability, would benefit from clarification to avoid potential misunderstandings regarding the nature of identifiability achieved. It appears that IDOL identifies the latent Markov Network rather than the true causal graph for the instantaneous component of the latent dynamics. This is an important distinction, as conditional independence relations allow only for the identification of the Markov equivalence class, not the directed causal structure itself. However, the presentation throughout the paper, especially in the introduction, experiments (such as Figure 4), and conclusions, may lead readers to infer that IDOL identifies the causal graph rather than the Markov network.\n\nTo address this issue, the authors could consider the following changes:\n\n- **Introduction (around line 89):** Indicate that the identifiability of the instantaneous structure in IDOL is only up to a Markov equivalence class, clarifying that IDOL does not identify the directions of edges in the instantaneous part.\n- **Figure 1c Modification:** Consider modifying Figure 1c to remove the arrow pointers from edges, signaling that the result is a Markov network rather than a causal graph when discussing identifiability (this might make sense in terms of theory, but not from a data generation perspective).\n- **Conclusion:** Mention the Markov equivalence class limitation explicitly. This would open a path for further research to extend the identifiability result from Markov equivalence to the full causal structure, especially given the promising empirical results observed in Figure 4.\n\nThe following specific statements in the theory section could be revised to improve clarity and accuracy:\n\n- lines 130-132: “the latent causal relations are also immediately identifiable because conditional independence relations fully characterize instantaneous causal relations in an instantaneous causally sufficient system”. I don’t think this line is correct without any additional assumptions. Conditional independence relations only provide the Markov equivalence class, not the exact causal graph, without further assumptions. Rephrasing this to accurately reflect the distinction between the Markov equivalence class and the causal graph would strengthen the theoretical foundation.\n- lines 171-172: Could you indicate whether $p_{c_t}$ refers to the marginal distribution $p(c_t)$ or the conditional distribution $p(c_t|z_{t-2})$?\nlines 165-188: For better readability, could you indicate $c_t \\in R^{2n}$ in your example? Otherwise, at first glance it reads as  $\\{z_{t,i}, z_{t-1,i} \\}$ for $c_{t,i}$ in Theorem 1.\n- line 217: Would it be better to use $\\emptyset$ to refer to $\\Phi$ as an empty set?\n- line 230: Could you define “isomorphic” for Markov networks? A footnote or reference to the Appendix suffices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ewcDAFd70M", "reviewer": "Reviewer_QmDF", "summary": "The paper presents a framework, IDOL (identification of instantaneous Latent Dynamics), for identifying temporally causal representation with instantaneous dependencies. IDOL employs a sparse latent process assumption, which is more adaptable to real-world data. The framework is validation through extensive experiments on both synthetic and rea-world human motion datasets.", "questions": "How does IDOL handle cases where the latent process sparsity assumption is only partially met? \n\nCould the authors clarify the computational complexity of IDOL compared to baselines, especially for high-dimensional data? \n\nAre there specific real-world scenarios where IDOL might struggle due to non-invertible mixing processes?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel approach to identifying temporally causal representations in time series data with instantaneous dependencies. This approach addresses a gap by proposing a sparse latent process assumption that is more practical for real-world applications than previous assumptions. \n\nExtensive evaluations are performance to demonstrate the effectiveness of the proposed approach.\n\nThe paper is well-organized. The use of illustrative figures helps clarify the complex concepts.", "weaknesses": "Providing further discussions on the possibility of extending IDOL to handle high-dimensional data can be beneficial. \n\nGiven the limitation due to the dependency on invertible mixing processes, providing guidelines for real-world applicability would add value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4GsFj5gOg", "reviewer": "Reviewer_uLFc", "summary": "This paper introduces a framework called IDOL (Identification framework for Instantaneous Latent dynamics) to enhance temporally causal representation learning for time series data with instantaneous dependencies. Traditional approaches for identifying latent causal processes in time series data often assume that the latent causal variables lack instantaneous interactions, limiting real-world applicability. IDOL addresses this limitation by applying a sparsity constraint on causal influences, allowing for both time-delayed and instantaneous dependencies in latent variables. The IDOL frameworkassumes a sparse influence within latent causal processes, allowing both time-delayed and instantaneous relations. Unlike prior methods that require data interventions or predefined groupings to achieve identifiability, IDOL elies on this sparse latent structure alone, making it highly applicable to scenarios where interventions are impractical. The framework’s theoretical foundation is built on leveraging sufficient variability and temporal contextual information, establishing identifiability through a combination of variational inference and sparsity regularization. This enables the model to accurately reconstruct latent variables and the underlying causal relationships without complex external assumptions.", "questions": "Can you please comment on the performance of your method in noisy environments and low-sample regimes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed IDOL framework moves beyond traditional methods that often rely on grouping of variables or direct interventions, by introducing a sparse influence assumption to capture the natural sparsity in many real-world datasets. This approach is novel in handling instantaneous dependencies without requiring interventions or grouping. Furthermore, the paper demonstrates rigorous theoretical and empirical quality, supported by a well-founded identifiability proof and a solid mathematical framework. Experimental validation on both synthetic and real-world human motion datasets further underscores the robustness and reliability of the model, showcasing its ability to accurately identify causal relationships and achieve high predictive accuracy is synthetic and real-world datasets. The paper is overall clearly written and easy to follow. Overall, this work is significant for the field, since causal discovery for time series with instantaneous effects is an important open problem.", "weaknesses": "The model assumes an invertible mixing process to reconstruct latent causal structures, which may not always be feasible in real-world data. In some scenarios, particularly in non-linear and noisy environments, this assumption could lead to inaccurate or incomplete latent representations, potentially undermining the model’s performance and causal interpretability. Furthermore, IDOL’s effectiveness heavily depends on the assumption of a sparse latent process. In cases where this sparsity assumption does not hold (i.e., when the causal structure is dense or complex), IDOL’s performance degrades, as demonstrated in the experiments. This sensitivity suggests that the framework may be less robust in scenarios where latent processes are highly interconnected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)