Temporally Disentangled Representation Learning
Authors: Weiran Yao
Guangyi Chen
Kun Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised learning
Disentanglement
Nonlinear ICA
Identifiability theory

Abstract: Recently in the field of unsupervised representation learning, strong identifiability results for disentanglement of causally-related latent variables have been established by exploiting certain side information, such as class labels, in addition to independence. However, most existing work is constrained by functional form assumptions such as independent sources or further with linear transitions, and distribution assumptions such as stationary, exponential family distribution. It is unknown whether the underlying latent variables and their causal relations are identifiable if they have arbitrary, nonparametric causal influences in between.  In this work, we establish the identifiability theories of nonparametric latent causal processes from their nonlinear mixtures under fixed temporal causal influences and analyze how distribution changes can further benefit the disentanglement. We propose TDRL, a principled framework to recover time-delayed latent causal variables and identify their relations from measured sequential data under stationary environments and under different distribution shifts. Specifically, the framework can factorize unknown distribution shifts into transition distribution changes under fixed and time-varying latent causal relations, and under global changes in observation. Through experiments, we show that time-delayed latent causal influences are reliably identified and that our approach considerably outperforms existing baselines that do not correctly exploit this modular representation of changes. 

Temporally Disentangled Representation Learning
Weiran Yao
Guangyi Chen CMU & MBZUAI
Kun Zhang CMU MBZUAI
CMU
wei weirannecmuedd
guangyichenl 994 @gmai com
kunz @ cmu edu
Abstract
Recently in the field of unsupervised representation learning, strong identifiability results for disentanglement of causally-related latent variables have been estab- lished by exploiting certain side information, such class labels, in addition independence. However, most existing work constrained by functional form assumptions such independent sources or further with linear transitions, and distribution assumptions such as stationary, exponential family distribution. It is unknown whether the underlying latent variables and their causal relations identifiable they have arbitrary, nonparametric causal influences in between. this work, establish the identifiability theories of nonparametrie latent causal processes from their nonlinear mixtures under fixed temporal causal influences and analyze how distribution changes can further benefit the disentanglement. We propose TDRL, a principled framework to recover time-delayed latent causal vari- ables and identify their relations from measured sequential data under stationary environments and under different distribution shifts. Specifically, the framework can factorize unknown distribution shifts into transition distribution changes under fixed and time-varye varying latent causal relations, and under observation changes in ob- servation. Through experiments, we show that time-delayed latent causal influences are reliably identified and that our approach considerably outperforms existing baselines that do not correctly exploit this modular representation of changes. Our code is available at: https: httptthhww..mm///eaaaaa////a//
Introduction
Causal reasoning for time-series data is fundamental task numerous fields II 2 Most existing work focuses on estimating the temporal causal relations among observed variables. However, in many real-world scenarios, the observed signals (e.g., image pixels in videos) do not have direct causal edges, but are generated by latent temporal processes confounders that causally related. Inspired by these scenarios, this work aims to uncover causally-relate latent processes and their relations from observed temporal variables. Estimating latent causal structure from observations, which we assume are unknown (but invertible) nonlinear mixtures of the latent processes, is very challenging. It has been found in 4 5 that without exploiting appropriate class of assumptions in estimation, the latent variables are not identifiable in the most general case. As a result, one cannot make causal claims on the recovered relations the latent space.
Recently, in the field unsupervised representation learning, strong identifiability results of latent variables have been established 6 7
& g 10] by using certain side information in nonlinear Independent Component Analysis (ICA), such as class labels, addition to independence. For time-series data, history information is widely used as the side information for the identifiability of latent processes. To establish identifiability, the existing approaches enforce different sets of functional and distributional form assumptions constraints in estimation; for example, (1) PCL [ and Slow VAE [2] assume indeeeeden sources in the
7. GCL [ HM-NLICA data generating process. However, this assumption may severely distort the identifiability if the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Attributes of nonlinear ICA theories for time-series. A check denotes that method has an attribute or can be applied to a setting, whereas a cross denotes the opposite. indicates our approach.
Time-varying Relation
Causally-related Partitioned
Nonparametric Transition
Applicable to Stationary Environment
Theory
Process
Subspace
PCL
X
X
X
GCL
X
x
HM-NLICA
X
K
X
X
Slow VAB
X
X
X
X
SNICA
V
X
X
X
i-VAE
X
X
X
X
LEAP
/
X
X
TDRL
latent variables have time-delayed causal relations in between (i.e., aausall--related process); (2) VAE [2] and SNICA [S] assume linear relations, which may distort the identifiability results if the underlying transitions nonlinear, and (3) SAowVAE [ assumes that the process noise is drawn from Laplacian distribution; i-VAE [9] assumes that the conditional transition distribution is part of the exponential family. However, in real-world scenarios,
cannot choose proper set of functional and distributional form assumptions without knowing in advance the parametric forms of the latent temporal processes. Our first step hence to understand under what conditions the latent causally processes are identifiable if they have nonparametric transitions in between. With the pro- posed condition, our approach allows recovery latent temporal causally-related processes in stationary environments without knowing their parametric forms in advance.
On the other hand, nonstation- arity has greatly improved the identifiability results for learn- ing the latent causal structure 14 15 16.. For instance, I.EAP
1=1,2.... T
Latent Causal Process Estimation
Change factors
14
established the identifiabil-
Gravity =
transition dynamics and observations
ity of latent temporal processes, but in limited nonstationary cases, under the condition that the dis- tribution of the noise terms of the latent processes varies across
Noise 0
Video domains
dyn
gbx
Mass 1.5
Gravity 10
Element
all segments. Our second step is to analyze how distribution shifts benefit stationary con- dition and to extend our con- dition to general nonstation- ary case. Accordingly, our ap- proach enables the recovery of la- tent temporal causal processes in general nonstationary environ- ment with time-vary varyyng relations such as changes in the influencing strength or switching some edges off 17 over time or domains.
Noise
2 2
X
Mass =
9
Gravity =
Transifions
Noise 0.1
Time iimeddeiyyed Information
Time
Figure 1: TDRL: Temporally Disentangled Representation Learning. We exploit fixed causal dynamics and distribution changes from changing causal influences and global observation changes to identify the underlying causal processes. &,, is the
estimated latent process. 0."" is the change factor for transition dynamics, i.e., representing mass and gravity in this example.
Given the identifiability results, we propose a learning framework, called TDRL, to recover nonparametric time-delayed latent causal variables and identify their relations from measured temporal data under stationary environments and under nonstationary environments in which it is unknown in advance how the joint distribution changes across domains (we define it as "unknown distribution shifts"). For instance, Fig. [ shows an example of multiple video domains of a physical system under different mass, gravity, and environment rendering settings With TDRL
0, is the change factor for observation, i.e., noise scale.
the differences across segments are characterized by the learned change factors @dhn of domain (note that domain index is given the model) that encode changes in transition dynamics, and changes in observation styles modeled by 0,h0 (we use "causal dynamics" and "latent causal relations/infuuencee" interchangeably). We then present generalized time-series data generative model that takes these change factors arguments for modeling the distribution changes. Specifically,
The variables and functions with "hat" estimated by the model: the without "hat" ground truth.

---

we factorize unknown distribution shifts into transition distribution changes in stationary processes, time-varying latent causal relations, and global changes in observation by constructing partitioned latent subspaces, and propose provable conditions under which nonparametric latent causal processes can be identified from their nonlinear invertible mixtures. We demonstrate through number of real- world datasets, including video and motion capture data, that time-delayed latent causal influences are reliably identified from observed variables under stationary environments and unknown distribution shifts. Through experiments, show that our approach considerably outperforms existing baselines that do not correctly leverage this modular representation of changes.
Related Work
Causal Discovery from Time Series Inferring the causal structure from time-series data is critical to many fields including machine learning (1. econometrics (2, and neuroscience [3] Most existing work focuses on estimating the temporal causal relations between observed variables. For this task, constraint- methods [8] apply the conditional independence tests to recover the causal structures, while score-based methods [T9 20 define score functions to guide a search process. Furthermore, [21 22
propose to fuse both conditional independence tests and score-based methods.
The Granger causality 23 and its nonlinear variations
24 25
are also widely used.
Nonlinear ICA for Time Series Temporal structure and nonstationarities were recently used to achieve identifiability in nonlinear ICA. contrastte learning (TCL [6]) used the independent sources assumption and leveraged sufficient variability in variance terms of different data segments. Permutation-based contrastive (PCL
7 proposed a learning framework which discriminates be- tween true independent sources and permuted ones, and identifiable under the uniformly dependent assumption. HM-NLICA [I] combined nonlinear ICA with a Hidden Markov Model (HMM) to au- tomatically model nonstationarity without manual data segmentation. i- VAE O introduced VAEs to approximate the true joint distribution over observed and auxiliary nonstationary regimes. Their work assumes that the conditional distribution is within exponential familics to achieve the identifiability of the latent space. The most recent literature nonlinear ICA for time-series includes LEAP [14 and (i-)CITRiS 26. 127] LEAP proposed nonparametric condition leveraging the nonstationary noise terms. However, all latent processes changed across contexts and the distribution changes need to be modeled by nonstationary noise and it does not exploit the stationary nonparametric components for identifiability. Alternatively, CITRIS proposed for identification of scalar and multidimensional latent causal factors. This approach does not suffer from functional distributional form constraints, but needs access active intervention.
intervention target information
3
Problem Formulation
3.1 Time Series Generative Model
Stationary Model As fundamental case, first present regular, stationary time-series generative process where the observations x, comes from nonlinear (but invertible) mixing function that maps the time-delayed causally-related latent variables z₁ to x₃. The latent variables or processes z, have stationary, nonparamettic time-delayed causal relations. Let T be the time lag:
X = g((())
Zit = Jt (1=)t-113,.-- E Pa(za)); Est) with es Pes
Nonlinear mixing
Stationary nonparametric transition
Stationary noise
Note that with nonparametric causal transitions, the noise term est ~ Pc; (where Pr, denotes the distribution of eg)) and the time-delayed parents Pa(zu) of zu (i.e., the set of latent factors that directly cause 25c) are interacted and transformed in an arbitrarily nonlinear way to generate 2ạt. Under stationarity assumptions, the mixing function g. the transition functions f. and the noise distributions Pe, are invariant. Finally, we assume that the noise terms are nutually-independenl (i.e., spatially and temporally independent), which implies that instantaneous causal influence between latent causal processes is not allowed by the formulation. The stationary time-series model in the fundamental case is used to establish the identifiability results under fixed causal dynamics in Section 4.
Nonstationary Model We further consider two violations of the stationarity assumptions in the
fundamental case, which lead to two nonstationary time series models. Let u denote the domain
3

---

or regime index. Suppose there exist m regimes of data, i.e,, u, with r = 1.2.... m, with un- known distribution shifts. In practice, the changing parameters of the joint distribution across domains often lie in low-dimensional manifold (28) Moreover, if the distribution is causally factorized, the distributions are often changed in a minimal and sparse way [29]. Based on these
assumptions, we introduce the dilddimeniooal minimal change factor (6 gehs), which was proposed in [30], to respectively capture distribution shifts in transition functions and observation. The vector 0,- = (04m ) has a constant value in each domain but varies across domains. The formulation of the nonstationary time-series model is in line with [0)] The nonstationary model is used to establish the identifiability results under nonstationary cases in Section 4.2 where show that the violation of stationarity in both ways can even further improve the identifiability results. We first present the two nonstationary cases. (1) Changing Causal Dynamics. The causal influences between the latent temporal processes are changed across domains in this set- ting. We model it by adding the transition change factors @ as inputs to the transition function: Zie = fi ((2,,--123.e-r E Pa(zu)) e Cu)) (2) Global Observation Changes. The global properties of the time series (e.g., video styles) are changed across domains in this setting. Our model captures them using latent variables that represent global styles; these latent variables are generated by a bijection Sa that transforms the noise terms 6i,t into the latent with change factor 0:th Fi. = (h e..)) Finally, we can deal with a more general nonstationary case by combining the three types of latent processes in the latent space in modular way. (3) Modular Distribution Shifts. The latent space has three blocks
Z = (z," chę _x( where zfxx 2,,t is the 9th component of the fixed dynamics parts, = is the ethh component of the changing dynamics parts, and z is the th component of the observation changes. The functions [f fol capture fixed and changing transitions and observation changes for each dimension of z₁ in Eq.
_ix
fe ((zi,er) E Pa(z..)),,..
2a,t
fo (2..--1 Pa(= }) 0th Fee))
(1)
So (0tth, En,e)
X, g(z)).
3.2 Identifiability of Latent Causal Processes and Time-Delayed Latent Causal Relations
We define the identifiability of time-delayed latent causal processes in the representation function space in Definition 1. Furthermore, if the estimated latent processes can be identified at least up to permutation and component-win wise invertible nonlinearities, the latent causal relations are also immediately identifiable because conditional independence relations fully characterize time-delayed causal relations in time-delayed causally sufficient system, in which there are no latent causal confounders in the (latent) causal processes. Note that invertible component-wise transformations on latent causal processes do not change their conditional independence relations.
Definition (Identifiable Latent Causal Processes). Formally let {x.) be sequence of observed variables generated by the true temporally causal latent processes specified by (Ja, ((e,((e)) g) given in Eq. 1 A learned generative model (fr, p(E:), g) is observationally equivalent to (fi, Or, (Ex), g) if the model distribution Pj.0....g (1x,)= matches the data distribution Ps....p..g ((xi) (T=1) everywhere. We say latent causal processes are identifiable if observational equivalence can lead to identifiability of the latent variables up to permutation and component invertible transformation T:
Pj. Br & (1x) = Ps. P., (fx,)?- - -(xx) = -ooooT VX, EXEE X
(2)
where X is the observation space.
4 Identifiability Theory
We establish the identifiability theory of nonparametric time-delayed latent causal processes under three different types of distribution shifts. W.L.o.g., we consider the latent processes with maximum time lag L, 1. The extentions to arbitrary time lags are discussed in Appendix ST.55 Let k be the element index of the latent space 2, and the latent size is n. In particular, (1) under fixed temporal causal influences, we leverage the distribution changes p( a 2 for different values

---

of z5-15 (2) when the underlying causal relations change over time, we exploit the changing causal influences on p(zk,t/ 1 u,) under different domain u,, and (3) under global observation changes, the nonstationarity p(za,e |ur) under different values of u, is exploited. The proofs are provided in Appendix ST
The comparisons between existing theories are in Appendix ST.33
4.1
Identifiability under Fixed Temporal Causal Influence
Let lik A log (= Assume that Tkk is twice differentiable in =k, and is differentiable in 2,,-11 = 1,2, ..... n. Note that the parents of 2g,1 may be only a subset of z₁-1 21,-1 is not a parent of kn,r, then Dmaa = O. Below we provide a sufficient condition for the identifiability of zr, 02l
followed by a discussion of specific unidentifiable and identifiable cases to illustrate how general it is. Theorem (Identifiablity under a Fixed Temporal Causal Model). Suppose there exists invertible function g that maps to 2t, i.e.,
2. = (x,)
such that the components of 2 are mutually independent conditional on 21-1. Let
Vi.t
8 BELLES
o Mut 8o2. B
8
Va.t
due DE BEL.E1
nut D
If for each value of 2t. Vi.:; Vids V2,1, V2,t.... Vn,t. Vanto 2n vector functioon in 21,t- 1, 22,t Fn,t-1, are linearly independent, then Ze must be an invertible, component-we transformation of permuted version of 2t-
The linear independence condition in Theorem is the core condition to guarantee the identifiability of &, from the observed X5. To make this condition more intuitive, below we consider specific unidentifiable cases, in which there is no temporal dependence in 2, the noise terms in 2, additive Gaussian, and two identifiable cases, in which 2, has additive, heterogeneous noise or follows some linear, non- Gan-Gassiian temporal process.
Let us start with two unidentifiable cases. In case N1, t; is an independent and identically distributed (i..d.) process, i.e., there is causal influence from any component of z to any 2k.t- In this case, Vk.. and VE., (defined in Eq. 4 are always O for A = 1,2, 2... n, since p(2), 22-1) does not involve 2 So the linear independence condition is violated. In fact, this is the regular nonlinear ICA problem with i.i.d. data, and it is well-known that the underlying independent variables are not identifiable [5). In case N2, all z follow additive noise model with Gaussian noise terms, i.e.,
(5) where is transformation and the components of the Gaussian vector e, are independent and 8' also independent from 24-1- Then 32me is constant, and 0z,,00 azg
(( Et.
violating the linear independence condition. In the following proposition we give some alternative solutions and verify the unidentifiability in this case.
Proposition (Unidentifiability under Gaussian Noise). Suppose =((z)) was generated by Eq. 5 where the components of et are mutually independent Gaussian and also independent from where D is arbitrary non-singular diagonal matrix, U is an arbitrary orthogonal matrix, and D2 is a diagonal matrix with Var 1/2 (/((,))))) as its w diagonal entry, a valid solution to satisfy the condition that the components of 2, are mutually independent conditional 2:-1-
2--1- Then any 2,- = DDDD2
Roughly speaking, for randomly chosen conditional density function p(=k.t 12-1) in which 2 is not independent from 2t-1 (i.e., there is temporal dependence in the latent processes) and which does not follow additive noise model with Gaussian noise, the chance for its specifie second- and third-order partial derivatives to be linearly dependent is slim. Now let us consider two cases in which the latent temporally processes 2, are naturally identifiable. First, consider case YI, where zn, follows a heterogeneous noise process, in which the noise variance depends on its parents:
Zket qu (Zt + by (24-1) Ek
(6)
Here we assume 6x,e is standard Gaussian and 61,0, 62,41 ... En mutually independent and inde- pendent from 2,-1- 1. which depends Z;-1, is the standard deviation of the noise in 24,t. (For
5

---

conciseness, we drop the argument of by and when there is no confusion.) Note that in this model, reduces to multiplicative noise model. The identifiability result of 24 is established in the following corollary.
if qu is 0 for all k = 1. ...,, n.
Corollary 1 (Identifiability under Heterogeneous Noise). Suppose x g(zz) was generated according to Eq. 6 and Eq. 3 hoolds true. bf on,
8bc and by- 88, abn (2k.1- 9u) bg in, 1' with
oqu
1, n, which are in total 2n function vectors in Zt-1. are linearly independent, then z, must be an invertible, component-wisg transformation of a permuted version of 24.
Let us then consider another special case, denoted by Y2, with a linear, non- Gaussian temporal model for Zy: the latent processes follow Eq. 5 with q being linear transformation and ER, following particular class of non-Gaussian distributions. The following corollary shows that 2, is identifiable as long as each z receives causal influences from some components of 2:-1-
Corollary 2 (Identifiability under a Specific Linear, Non-Gaussian Model for Latent Processes). Suppose g(zz) was generated according to Eq. 5 in which q is linear transformation and Dsk.
F O. Assume the noise term Ch,t for each 2g, there exists at least one k' such that Cr. J=0.e1
follows a zero-mean generalized normal distribution:
p((K,t) O 6 -Meaaall with positive X and B > 2 and 3 +
If Eq. holds, then 24 must be an invertible, component-wise transformation of permuted 24-
4.2 Further Benefits from Changing Causal Influences
Let (ur) - log (( Wp)) where r = 1,..., m. LEAP 14] established the identifiability of the latent temporal causal processes 2, in certain nonstationary cases, under the condition that the noise term in each ER,r. relative to its parents in - 1. changes across m contexts corresponding to u W1, U2, Um. Here show that the identifiability result shown in the previous section can further benefit from nonstationarity of the causal model, and that our identifiability condition is generally much weaker than that in [4]: we allow changes in the noise term or causal influence on Zk.t from its parents in Zt-1, and our "sufficient variability" condition is just a necessary condition for that in 14 because of the additional information that one can leverage. Let Vo,e(u,)) be Vk.t- which is defined in Eq. in the u, context. Similarly, Let (u,) be Vi,, in the Ur context. Let
Sk, (Ve,((u)),,... .... (Mm A2..... .... 4m)) $2.1 (Vr,)(u))T, ... Va.l(um) A2, 22, Am) (8)
where A2= (Pn))(u)) 8 nur 12 and A, (1.((1z) - Ouge( 1), As provided below, in our OzN.
0=2,,
O=u.1
case, the identifiablity of z, is guaranteed by the linear independence of the whole function vectors Sk,t and $2,t, with k- 1, 2.., n. However, the identifiability result in [44 relies on the linear independence of only the last m 1 components of s and $E.,, with k = 1,2, ...; n; this linear independence is generally much stronger and restricted condition.
Theorem 2 (Identifiability under Changing Causal Dynamics). Suppose = g(z,) and that the conditional distribution p(=K.t 2t-1) may change across m values of the context variable u, denoted by un, U2 Um- Suppose the components of z are mutually independent conditional on Zt-1 in each context. Assume that the components of 2 produced by Eq. Jare also mutually independent conditional on 2;-1- If the 2n function vectors Sx,1 and šg,t, with k = 1, 2..,,,,,, are linearly independent, then 2, is permuted invertible component-wii wise transformation of zt-
Theorem 3 (Identifiability under Observation Changes). Suppose x, g(z,) and that the conditional distribution p(za,1] u) may change across m. values of the context variable u, denoted by un, u2, Uan- Suppose the components of 24 are mutually independent conditional on in each context. Assume that the components of à, produced by Eq. 3 are also mutually independent conditional on 2e-1- If the 2n function vectors Sk,t and šk,t, with k =1,2,., n, are linearly independent, then 2, is a permuted invertible component-wise transformation of zt-
Corollary 3 (Identifiability under Modular Distribution Shifts), Assume the data generating process fx dig
in Eq. 1 If the three partitioned latent components conditions in Theorem 1, Theorem 2, and Theorem 3, then 24 must be an invertible, component-wise transformation of permuted version of 2,
(z 2z 2o05) respectively satisfy the

---

to output random process Our Approach
5.1 TDRL: Temporally Disentangled Representation Learning
Given our identifiability results, we propose TDRL framework to estimate the latent causal dy- namics under modular distribution shifts, by extending Sequential Variational Auto-Encoders 31 with tailored modules to model different distribution shifts, and enforcing the conditions Sec. as constraints. We give the estimation procedure of the latent causal dynamics model in Eq. The model architecture is showcased in Fig. - The framework has the following three major components. The implementation details are in Appendix |3.3 Specifically, we
leverage the partitioned estimated latent subspaces 2, = =(f," ,2, 22t and model their distri- bution changes in conditional transition priors. We use [fs, Sc: to capture causal relations as in Eq. [ where fs captures fixed causal influences, fe for changing causal influences, and fo for observation changes. Accordingly, we learn [ff= fc noise from the estimated direct cause (lagged states ZHx) and effect (current states) variables ž,.
f-1
f f.f.
Change Factor Representation
We learn to embed domain index u, into wow-diensiooal change factors
Fixed causal processes
Changing causal processes
Observation changes
(0, dym pbs
in Fig. 2 and insert them
A flx
Fixed fix
Z chg
f.
2
chg
gghs
f,(0
2
external inputs to the (inverse) dy-
&
namics function (0,", dyn, the ob-
Partitioned latent subspaces
servation bijector (0, respec- tively. And hence the distribution shifts are captured and utilized in the implementation.
Frame Encoder
Decoder
Reconstruction
Xt-L Xt-1 X
L & &
Modular Prior Network We fol- low standard conditional normaliz- ing flow formulation 32 [4]. Let ZHK denote the lagged latent vari- ables up to maximum time lag L.
Figure 2: TDRL describes each domain with change factors
dyn obs
(0, 0, and inserts them into the prior models the parti- tioned latent processes. The posteriors of the latent variables are inferred from image frames with variational autoencoder.
In particular, for 1) fixed causal dynamics processes 2t their transition priors are obtained by first learning inverse transition functions f-1 that take the estimated latent variables and output random noise terms, and applying the change of variables formula the transforma-
tion: p( sfx (zxx) Pe, 1 2Hx)
Øf- for 2) changing causal dynamics, we evaluate of.
dE
ghe p(=c,t BHx, u,) Pec (fa² (Ze,t .. 24k, 0,
dyn
by learning a holistic inverse dynamics f:
that takes the estimated change factors for dynamics dyy as inputs, and similarly for 3) global observation changes 20th, we learn to project them to invariant noise terms by f-1 which takes the change factors ge as arguments, and obtains p(2utr gobs (u)) = P.. (fo (2ote O,
af as the prior. Deahs
Conditional independence of the estimated latent variables p(2 (2нx) is enforced by summing up all estimated component densities when obtaining the joint plzel /žHx in Eq. 9 Given that the Jacobian lower-rringgula, we can compute its determinant as the product of diagonal terms. The detailed derivations are given in Appendix S3.1
log p (Zr 2Hx; u log p(E))u,))) +
log
Conditional indepdence
Lower triangular Jacobian
Factorized Inference We infer the posteriors of each time step q(,, using only the observation at that time step, because in Eq. [ X preserves all the information of the current system states so the joint probability q(arr] X1:r) can be factorized into product of these terms. We approximate the posterior q(2, xx) with an isotropic Gaussian with mean and variance from the inference network.
Optimization We train TDRL using the ELBO objective LELBO = Y EiEN LRecon 3L in which we mean-squared error (MSE) for the reconstruction likelihood LRecOn- The KL divergence
7

---

CLLD is estimated via sampling approach since with learned nonparametric modular transition prior, the distribution does not have an explicit form. Specifically, we obtain the log- likelihood of the posterior, evaluate the prior logp (2e Aisx Up) in Eq. 2 and compute their mean difference in the dataset as the KL. loss: LXLD
E.
2. ( px(0) log ( xx) - log p( [atx u,).
5.2 Causal Visualization
For visualization purposes, when the underlying latent processes have sparse causal relations, we fit LassoNet [31 on the latent processes recovered by TDRL to interpret the causal relations. Specifically, we fit LassoNet to predict 2, using the estimated history information 2Hx {2,- 1 up to maximum time lag L. Note that this postprocessing step is optional - the latent causal relations have already been captured in the learned transition functions in TDRL Also, our identifiability conditions do not rely on the sparsity of causal relations in the latent processes.
6 Experiments
We evaluate the identifiability results of TDRL on a number of simulated and real-world time-series datasets. We first introduce the evaluation metrics and baselines. (1) Evaluation Metrics. To evaluate the identifiability of the latent variables, we compute Mcan Correlation Coefficient (MCC) on the test dataset. MCC is standard metric in the ICA literature for continuous variables which measure the identifiability of the learned latent causal processes. MCC is close to 1 when latent variables are identifiable up permutation and component-wiss, invertible transformation in the noiseless case. (2) Baselines. Nonlinear ICA methods are used: (1) BetaVAE 624 which igmores both history and nonstationarity information; (2) iVAE O
and TCL 6 which leverage nonstationarity to establish identifiability but assumes independent factors, and (3) Slow
[2] and PCL 7 which exploit temporal constraints but assume independent sources and stationary processes, and (4) LEAP |E! which assumes nonstationary. causal processes but only models nonstationary noise. Two other disentangled deep stäte-space models with nonlinear dynamics models: Kalman (KVA 35
and Deep Variational Bayes Filters (DVBF (36) are also used for comparisons.
Table 2: MCC scores and their standard deviations for the three simulation settings over 3 random seeds. Note: The symbol "_" " represents that this method is not applicable to this dataset.
Experiment Settings
Method
LEAP
SlowVAE
PCL
i-VA
TCL
betaVAA
KVAE
DVBF
+0.009
0.832
0.778
Changing
0.958 10.017
0,726 40 187
0.511 20.062
0.599
10.041
0.581 +0.083
0,399 20.021
0,523 10.009
0,711 #0082
0.648 10.071
Modular
0.993 10.001
0,57 40 108
0,106 20.045
0.561 10.019
0.557 10.005
0.297 10.078
0:135 10.015
0.632 10.015
10.074
6.1 Simulated Experiments
We generate synthetic datasets that satisfy our identifiability conditions in the theorems following the procedures in Appendix [22.1.I As in Table 2 our framework can recover the latent processes under fixed dynamics (heterogeneous noise model), under changing causal dynamics, and under modular distribution shifts with high MCCs (>0.95). The baselines that do not exploit history (i.e., 3VAF, V-,, TCL.), with independent source assumptions (Slow VAF, PCL.), considers limited nonstationary cases (LEAP) distort the identifiability results. KVAE and DVBF achieve MCCs (0.8) under fixed dynamics but distorts the identifiability under changing dynamics and modular shift setting because they don't 't model changing causal relations and global observation changes.
6.2 Real-world Applications
Video Data Modified Cartpole Environment We evaluate TDRL on the modified cartpole K0 video dataset and compare the performances with the baselines. Modified Cartpole is nonlinear dynamical system with cart positions I; and pole angles 0r as the true state variables. The dataset descriptions are in Appendix 12. We use source domains with different gravity values g (5, 10, 15, 20, 25, 30). Together with the 2 discrete actions (i.e., left and right), we have 12 segments of data with changing causal dynamics. We fit TDRL with two-dimensional change factors 0dyn We set the latent size n and the lag number I. = 2. In Fig S the latent causal processes are recovered, as seen from (a) high MCC for the latent causal processes; (b) the latent factors are estimated up to
8

---

component-wiss transformation: (c) TDRL outperforms the baselines and (d) the latent traversals confirm the two recovered latent variables correspond the position and pole angle.
MCC=0.99 0,983
Cartpole
Estimated Latents
Estimated Lalents
TDRL
LEAP
KWAE
DARE
BetaWAE
WAE
0.09
0.99
Latents
Lstents
Steps
(a)
(b)
(c)
(d)
Figure 3: Modified Cartpole results: (a) MCC for causally-related factors; (b) scatterplots between estimated and true factors; (c) baseline comparisons, and (d) latent traversal on a fixed video frame;
Motion Capture Data CMU-Mocap We experimented with another real-world motion capture dataset (CMU-Mocap). The dataset descriptions are in Appendix S2.112 We fit TDRL with 11 trials of motion capture data for subject #8. The 11 trials contain walk cycles with very different styles and lag number I 2. The differences between
(e.g., slow walk, stride). We set latent size trials are captured through learning the 2-dimensional change factors for each trial. In Fig. a). the learned change factors group similar walk styles into clusters; in Panel (c), three latent variables (which seem to be pitch, yaw, roll rotations) are found to explain most of the variances of human walk cycles. The learned latent coordinates (Panel b) show smooth cyclic patterns with differences among different walking styles. For the discovered skeleton (Panel d), roll and pitch of walking are found to be causally-relee relatte while yaw has independent dynamics.
Change factors
Estimated Skeleton
Pitch
Rotation
2
v
Roll z
X
Away
2-1
7--2 P
(a) Domain embeddings
(b) Latent dynamics coordinales
(c) Latent traversal
(d) Skeleton. Blue means dependency
Figure 4: CMU-Mocap results (Subject #8): (a) learned change factors; (b) latent coordinates dynamics for trials; (c) latent traversal by rendering the reconstructed point clouds into the video frame; (d) estimated causal skeleton (blue indicates dependency).
7 Conclusion
In this paper, without relying on parametric or distribution assumptions, we established identifiability theories for nonparametric latent causal processes from their observed nonlinear mixtures in stationary environments and under unknown distribution shifts. The basic limitation this work is that the underlying latent processes are assumed to have no instantaneous causal relations but only time- delayed influences. If the time resolution of the observed time series is the same as the causal frequency (or even higher), this assumption naturally holds. However, if the resolution of the time series is much lower, then it is usually violated and one has to find way to deal with instantaneous causal relations. On the other hand, it is worth mentioning that the assumptions are generally testable; if the assumptions are actually violated, one can see that the results produced by our method may not be reliable. Extending our theories and framework address issue instantaneous dependency or instantaneous causal relations will be one line of our future work. In addition, Empirically exploration of the merits of the identified latent causal graph in terms of few-shot transfer to new environments 33 40 and control [30 is also one important future step.
37 domain adaptation
38 forecasting

---

Acknowledgment
Kun Zhang was partially supported by the National Institutes of Health (NIH) under Contract 201HL119885, by the NSF-Convergence Accelerator Track-D award #2134901, by a grant from Apple Inc., and by grant from KDDI Research Inc.
References
[1] Carlo Berzuini, Philip Dawid, and Luisa Bernardinell. Causality: Statistical perspectives and applications. John Wiley & Sons, 2012.
[2] Eric Ghysels, Jonathan B Hill, and Kaiji Motegi. Testing for granger causality with mixed frequency data. Journal of Econometrics, (92(1))207-230 2016.
[3] Karl Friston. Causal modelling and brain connectivity in functional magnetic resonance imaging. PLoS biology, 7(2)=10000333, 2009.
[4] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[5] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-44 439, 1999.
[6] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
[7] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460 469. PMLR, 2017.
[8] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[9] llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[10] Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-ffov networks (gin). arXiv preprint (00::20000048777 2020.
[11] Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty Artificial Intelligence, pages 939-948. PMLR, 2020.
[12] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007. 10930, 2020.
[13] Hermanni Hälvä, Sylvain Le Corff, Lue Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. arXiv preprint ar arXiv:2106.09622 2021.
[14] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. International Conference on Learning Representations, 2022.
[15] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint 10111:1901.10912, 2019.
10

---

[16] Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer. Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv: 19000109100 2019.
[17] Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. arXiv preprint arXiv:2007.0063/ 2020.
[18] Doris Entner and Patrik O Hoyer. On causal discovery from time series data using fci. Proba- bilistic graphical models, pages 121-128, 2010.
[19] Kevin Murphy et al. Dynamic bayesian networks. Probabilistic Graphical Models, M. Jordan, 7:431, 2002.
[20] Roxana Pamfil, Nisara Sriwattanaworrcchii, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference on Artificial Intelligence and Statistics, pages 1595-1605. PMLR, 2020.
[21] Daniel Malinsky and Peter Spirtes. Causal structure learning from multivariate time series in settings with unmeasured confounding. In Proceedings of 2018 ACM SIGKDD Workshop on Causal Discovery, pages 23-47. PMLR, 2018.
[22] Daniel Malinsky and Peter Spirtes. Learning the structure of a nonstationary vector autore- gression. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2986-2994. PMLR, 2019.
[23] Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pages 424 438, 1969.
[24] Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily Fox. Neural granger causality. arXiv preprint arXiv: 1802.88044.4 2018
[25] Sindy Löwe, David Madras, Richard Zemel, and Max Welling, Amortized causal discovery: Learning to infer causal graphs from time-series data, arXiv preprint arXiv:2006. 10833, 2020.
[26] Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022.
[27] Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. icitris: Causal representation learning for instantancous temporal effects. arXiv preprint arXiv:2206.061699 2022.
[28] Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang, Data-driven approach to multiple-source domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statisties, pages 3487-3496. PMLR, 2019.
[29] AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning in linear systems. Advances in neural information processing systems, 31, 2018.
[30] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021.
[31] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv preprint arXiv: 1803.02990 2018.
[32] Laurent Dinh, Jascha Sohl- Sock-tiicstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605.00600.88800 2016.
[33] Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, 22(127):1-29, 2021.
11

---

[34] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
[35] Marco Fraccaro, Simon Kamronn, UIrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. Advances in neural information processing systems, 30, 2017.
[36] Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432. 2016.
[37] Weiran Yao, Guangyi Chen, and Kun Zhang. Learning latent causal dynamics. arXiv preprint arXin:2202.04828, 2022.
[38] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, edi- tors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 11455-11472. PMLR, 17-23 Jul 2022.
[39] Biwei Huang, Kun Zhang, Mingming Gong, and Clark Glymour. Causal discovery and forecasting in nonstationary environments with state-space models. In International conference on machine learning, pages 2901-2910 PMLR, 2019.
[40] Zijian Li, Ruichu Cai, Tom ZJ Fu, and Kun Zhang. Transferable time-series forecasting under causal conditional shift. arXiv preprint arXiv:2111.0342 2021.
[41] Alessio Spantini, Daniele Bigoni, and Youssef M. Marzouk. Inference via low-dimensional couplings. J. Mach. Learn. Res., 19:66:1 1666:1-660 1-6:71, 2018.
[42] Kiyotoshi Matsuoka, Masahiro Ohoya, and Mitsuru Kawamoto. A neural net for blind separation of nonstationary signals. Neural networks, 8(3):11-419, 1995.
[43] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambriiggeUnivverittyrersss 19, 2000.
[44] Hans Reichenbach. The direction of time, volume 65. Univ of California Press, 1956.
[45] Clive WJ Granger. Implications of aggregation with common factors. Econometric Theory, (02):2208 222, 1987.
[46] M. Gong*, K. Zhang*, D. Tao, Geiger, and B. Schölkopf. Discovering temporal causal relations from subsampled data. In Proc. 32th International Conference on Machine Learning (ICML 2015), 2015.
[47] D. Danks and S. Plis. Learning causal structure from undersampled time series. In JMLR: Workshop and Conference Proceedings, pages 1-10, 2013.
[48] M. Gong, K. Zhang, Schölkopf. C. Glymour, and D. Tao. Causal discovery from temporally aggregated time series. In Proc. Conference on Uncertainty in Artificial Intelligence (UAI'17), 2017.
[49] R. Silva, Scheines, C. Glymour, and Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7:191-246, 2006.
[50] J. Adams, N. R. Hansen, and K. Zhang. Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
12


Reviews:
[{"review_id": "sz_UjR2M5Of", "reviewer": "Reviewer_nJM9", "summary": "This paper proposes a formulation to recover latent temporal processes from observed time series data. Assumptions for identifability of the latent processes relax the ones posited in previous works and now rely on linear independence of specific vectors of partial derivatives of the latent update function. Authors also consider the case with observations from different domains and with time varying causal effects between hidden processes. The authors then propose an architecture enforcing the specific linear independence assumption and show that they can identify the latent temporal processes for an artificially generated and cart pole dataset. ", "questions": "- What are the exact conditions on g(.) ? Should it be an invertible function ? \n- Enforcing conditions of Eq 4. With prior model and KL loss. Use sampling to infer KL divergence. How do you make sure that this results in satisfying conditions of Eq.4 ? More generally, how do you make evaluate if your data and model satisfies your assumption in the end ? Was there any issue in the training and in getting the KL loss low ?\n- How does the sampling for the KL loss impacts computational considerations. How does this scale for larger number of latents and number of lags?\n- How do you divided the number of latents for each group (fixed,c,o) ? What happens when the number of latents is misspecified ? It seems you use n=8 in the cart pole but the true number of latents is 2. How do you select the right ones ?", "limitations": "I think the assumptions of the model are clear in general, except for the read-out function g(.) (see above).\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n\n- The problem addressed is important and builds upon solid previous work.\n- The work relaxes distributional assumptions from previous works and provides identifiability proof.\n- The reported performance are clearly above other compared methods.\n\nWeaknesses : \n\n- There are some clarifications needed that I listed as questions below.\n- Why didn’t you evaluate the MCC of other baselines on the cart pole dataset ? This seems a more informative experiment for comparing the methods than the synthetic data that exactly matches your data generating assumption.\n- It would be great to have a formulation of the identifiability criteria for arbitrary lags. And understand how it impacts training.\n- Overall I’m not really convinced by the causal taxonomy of the model. Some assumptions are posited but it does not guarantee any causal discovery mechanism. The model formulation doesn't also allow to draw any causal conclusion between observables. In particular, it feels that the action in the cart pole data is modelled as a different set of parameters which thus don’t really model the treatment effect in the observed process. \n\nMinor\n- C is not shown on Figure 2\n- Some typos : \n    - Line 49 : question arises *that* (does not sound really correct to me)\n    - Line 185 : So *th* linear independence (« the » ?)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gSYBPO1SgR", "reviewer": "Reviewer_U8Ti", "summary": "This paper considered establishing identifiability theories for nonparametric latent causal processes from their observed nonlinear mixtures. The authors proposed to recover time-delayed latent causal variables and identify their relations from measured temporal data, namely LiLY. The theoretical results were derived for both stationary environments and unknown distribution shifts, with promising results in real-world applications.", "questions": "Please check the questions listed in the Weaknesses above. Any justifications are welcomed.", "limitations": "I didn't find any limitations discussed. Please correct me if I am missing anything.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\n - This paper handles an interesting problem in machine learning by considering the identifiability of causal dynamics in time series data. \n - The theoretical analyses are sound with promising real data analysis results.\n - This paper is well written and organized.\n\n*Weaknesses:*\n\n- The theoretical performance of the proposed LiLY for learning latent causal dynamics was not derived and discussed. Can the authors provide any theoretical analysis on this?\n- The partitioned latent subspace is pre-specified and limited to three components. Can the authors provide more justification for this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YSVrJPJzFtt", "reviewer": "Reviewer_7swA", "summary": "The paper proposes a new framework for nonlinear Independent Component Analysis, LiLY, which recovers time-delayed latent causal variables from temporal data under stationary environments and under different distribution shifts. The authors provide proofs for the identifiability of the causal variables in the settings under consideration and show experimentally that their approach achieves improved performance on data that matches their assumptions.", "questions": "A number of questions regarding this paper are embedded in the \"Strengths and Weaknesses\" section above. Besides these questions, I would be interested in a more concrete connection between the proposed approach and causal discovery. If the authors can answer these points convincingly, as well as outline how and where they would embed their answers into the paper, I will consider raising my score.", "limitations": "I could not find any discussion on the limitations of the proposed method, and would appreciate if the authors could highlight some limitations in their rebuttal.\n\n**Given the author's reply and revised version of the paper, I have increased my rating for this paper.**", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The proposed framework consideres a novel setting for nonlinear ICA and provides convincing identiability results in terms of theory and practice. \n\n- Overall, the paper follows a coherent structure and uses clear formulations for all theoretical and experimental settings.\n\n- Table 1 gives a great overview that summarizes the novelty of the proposed framework compared to existing research.\n\n# Weaknesses\n1) In my eyes, the biggest weakness of this paper is that it does not provide any convincing motivation for the proposed approach. In the current form, the paper argues that existing approaches have a certain set of assumptions and that the proposed approach uses different assumptions. However, this is not really an argument in itself. What I would like to know is: why do these assumptions matter; how are existing approaches limited by the assumptions that they use and what new applications do you enable with your new approach? To embed these answers into the paper, I would propose a rewrite of the introduction. Additionally, it would be nice to include a discussion on the assumptions in section 3.1, i.e. why they are needed and how restrictive they are in practice. Besides this, the paper is also missing a discussion on the proposed separation of distribution changes into three types. Why did you choose these three types; are there any alternative separations that you considered and why is the proposed solution the best; are there any distribution changes that these types cannot (jointly) represent? Finally, a discussion on the limitations of the proposed approach is missing. Overall, this makes it very difficult to judge the significance of this work.\n\n2) The connection between this method and causal discovery seems somewhat hand-wavy. The abstract claims that the proposed framework recovers latent variables and identifies their relations. While the experiments and theoretical results support the former claim, the latter seems to be missing from the paper, effectively removing the connection to causal discovery. Additionally, \"causal dynamics\" are mentioned throughout the paper without ever being defined.\n\n3) According to l.297, all evaluations were conducted on the validation datasets. This is not appropriate according to standard machine learning practice. For convincing results, these experiments need to be repeated on the test datasets.\n\n4) The paper's clarity could be improved in certain regards. For example, I found it confusing which variables and functions are estimated by the model and which ones are ground-truth factors. Does the model use the ground-truth transition functions, or does it estimate them? Does the model work with unknown distribution shifts as claimed in the abstract and intro, or does it require a domain index (l. 281) as input? \n\n\n\n## Others\n- Currently, the introduction also functions as a related works section. I would prefer a separation of these two sections. For one, this would allow the introduction to focus on the motivation for the proposed approach, as mentioned above. Additionally, a separate related work section could discuss existing work in more detail and provide an embedding of the proposed work into the existing research landscape - an aspect that is largely missing in the current form.\n\n- There are grammatical errors and spelling mistakes throughout the paper that a standard spell checker should be able to catch.\n\n- Personally, I am not a big fan of the name of the method, as well as the resulting acronym. The name \"Learning latent causal dynamics\" is too general, could be applied to a whole range of methods and does not describe the essence of the proposed method, and the acronym is gathering scattered letters with non-matching capitalization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6MDkqMfQBg", "reviewer": "Reviewer_BcGf", "summary": "This paper proposes a novel method for identifying causal variables from time-series data with time delayed effects. In order to do so, the authors consider data recorded from different distribution shifts/environments. The latent variables are grouped into three main parts: a dynamics part that is fixed across all environments, a dynamics part changing across environments, and finally global changes to the observation space. The paper discusses theoretical results and a practical implementation called LiLY, which outperforms previous methods on a set of synthetic and real-world inspired datasets.", "questions": "* Can you provide a clearer comparison of your method to [Huang et al., 2022]?\n* What are the limitations of your method from your perspective?", "limitations": "As mentioned in the \"Weakness\" section, no limitations have been discussed, as well as no potential negative social impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper proposes an interesting approach to identifying latent causal variables from temporal data. It covers more settings than previous work and supports different transition dynamics and observation noises. The overall approach towards identifiability is clear and closely follows previous works on non-linear ICA. I especially appreciate the thorough discussion in Section 3.1 of various settings which could be identified only with or without the linear independence condition. The theory seems sound, although I have not checked the proofs in the Appendix in detail. The proposed implementation model supposedly supports non-trivial distributions by using Normalizing Flows in the VAE prior. Finally, the experiments cover several settings well in which the proposed model is compared to a set of relevant baselines and shows to outperform previous methods.\n\n## Weaknesses\n\nThe clarity of the paper is adversely affected by a confusing notation in several places. Specifically, the following notation was not fully introduced before usage:\n* In lines 155-159, the latent space $z$ is indexed by $k$, which had been introduced in Section 2 as a number over domains ($k=1,...,K$) and index of $u$. However, here it seems to be running over different values without explanation.\n* Eq 1: the variable \"$\\tau$\" is undefined\n* Line 168: the variable \"$n$\" is undefined\n* Between line 224-225 (line numbering issue): the index $r$ as well as Eq. 11 are not in the main text\n* Between line 224-225: it is unclear why the different environments are now indexed with $1,...,m$ while previously used $1,...,K$\n* Line 225: $\\eta_{kt}$ had been defined as $\\log p(z_{k,t}|z_{t-1})$, unclear how it is now a function of $u$\n* Line 256-260: \"$k$\" is here again used for domains instead of latent variables\nIt is strongly recommended to keep the notation consistent throughout the paper, since it is currently difficult to fully understand all statements when the notation is unclear.\n\nA second weakness of the paper is that the comparison to one of the most related works, [Huang et al., 2022] (cited as [33] in the paper), is very sparse. Section 2 beings with 'Following [32,33], ...'. It is unclear what parts are taken from [Huang et al., 2022], and which parts are own definitions/setting descriptions. A more direct discussions of the similarities between this work and [Huang et al., 2022] should be stated. Furthermore, I would have expected a similar discussion in the model part, Section 4, since both follow a similar strategy with an observation setting $\\theta^{\\text{obs}}$ and dynamics setting $\\theta^{\\text{dyn}}$.\n\nAnother weakness of this work is that it does not discuss the limitations of the work, despite the authors stating so in the paper checklist. Every work on identifiability has its limitations, since certain assumptions need to be taken. However, it is important to address when these assumptions might hold in a realistic setting, and what pre-knowledge was used in creating the models. For instance, Section 4 suggests that in order to obtain identifiability, one needs to exactly know how many variables there are for all three parts: static dynamics, changing dynamics, and observation. This seems quite limiting, since one likely can only estimate those in a real-world setup. How sensitive is the method to this? What happens when too large sets are used? How sensitive is the model to hyperparameters? A fair discussion of other limitations would also be needed. For example, in a real-world system where we do not know the causal variables, how can one ensure that the assumption of Theorem 1 holds (i.e. the $2n$ vector functions being linearly independent)? \n\nFinally, while the experiments cover a decent range of settings, all of them have a rather simple observation space. To strengthen the paper, it is beneficial to include an experiment on a visually complex dataset, for example, Causal3DIdent [von Kügelgen et al., 2021; Zimmermann et al., 2021], more specifically its extension to temporally data [Lippe et al., 2022]. This would highlight the method's capability of identifying causal variables in complex settings.\n\n\n### Minor points\n\n* Figure 2: the noise variables of the observation, $\\hat{\\epsilon}\\_{i,t}$, are labeled as time-delayed information, although they have the same index and are likely not delayed information.\n* The paragraph about the factor representation (line 280-283) comes a bit late, since these have been already used in the previous paragraphs. It would be easier to state this up front.\n\n### Typos\n\n* Line 101: space before comma - \"...domains , ...\"\n* Line 112: \"Let u denote the...\" (no s)\n* Line 184: dot after 1 - \"k=1.,2,...\"\n* Line 185: incomplete word - \"So th linear...\"\n* Line 232: time index $t$ missing - $z_{-1}$\n* Line 268: \")\" too much\n* Line 296: capital \"We\" within the sentence\n\n### Conclusion\n\nThe proposed method is interesting and potentially relevant for researchers in causal representation learning. The current version lacks a bit in clarity and misses a fair discussion of its limitations. Still, I believe that these points can be addressed in a revised version/the rebuttal. Hence, my current score is \"Weak Accept\".\n\n### References\n\n[Huang et al., 2022] Huang, Biwei, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. 2022. AdaRL: What, where, and how to adapt in transfer reinforcement learning. International Conference on Learning Representation (ICLR).\n\n[von Kügelgen et al., 2021] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. 2021. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. In Thirty-Fifth Conference on Neural Information Processing Systems.\n\n[Zimmermann et al., 2021] Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. 2021. Contrastive Learning Inverts the Data Generating Process. In Proceedings of the 38th International Conference on Machine Learning.\n\n[Lippe et al., 2022] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. 2022. CITRIS: Causal Identifiability from Temporal Intervened Sequences. In Proceedings of the 39th International Conference on Machine Learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept