Learning Temporally Causal Latent Processes from General Temporal Data
Authors: Weiran Yao
Yuewen Sun
Alex Ho
Changyin Sun
Kun Zhang
Venue: ICLR 2022 Poster (2022)

Abstract: Our goal is to recover time-delayed latent causal variables and identify their relations from measured temporal data. Estimating causally-related latent variables from observations is particularly challenging as the latent variables are not uniquely recoverable in the most general case. In this work, we consider both a nonparametric, nonstationary setting and a parametric setting for the latent processes and propose two provable conditions under which temporally causal latent processes can be identified from their nonlinear mixtures. We propose LEAP, a theoretically-grounded framework that extends Variational AutoEncoders (VAEs) by enforcing our conditions through proper constraints in causal process prior. Experimental results on various datasets demonstrate that temporally causal latent processes are reliably identified from observed variables under different dependency structures and that our approach considerably outperforms baselines that do not properly leverage history or nonstationarity information. This demonstrates that using temporal information to learn latent processes from their invertible nonlinear mixtures in an unsupervised manner, for which we believe our work is one of the first, seems promising even without sparsity or minimality assumptions. 

Published as a conference paper at ICLR 2022
LEARNING TEMPORALLY CAUSAL LATENT PRO- CESSES FROM GENERAL TEMPORAL DATA
Weiran Yao Yuewen Sun
Alex Ho
Changyin Sun
Kun Zhangi
i CCrnegie Mellon University, Pittsburgh PA, USA
i "Souhheast University, Nanjing, China
'Rice University, Houston TX, USA
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
ABSTRACT
Our goal is to recover time-delayed latent causal variables and identify their re- lations from measured temporal data. Estimating causally-related latent vari- ables from observations is particularly challenging as the latent variables are not uniquely recoverable in the most general case. In this work, we consider both a nonparametric, nonstationary setting and parametric setting for the latent pro- cesses and propose two provable conditions under which temporally causal latent processes can be identified from their nonlinear mixtures. We propose LEAP, a eeooreticallyggrunnee
framework that extends Variational AutoEncoders (VAEs) by enforeing our conditions through proper constraints in causal process prior. Experimental results on various datasets demonstrate that temporally causal latent processes are reliably identified from observed variables under different depen- dency structures and that approach considerably outperforms baselines that do not properly leverage history or nonstationarity information. This demonstrates that using temporal information learn latent processes from their invertible non- linear mixtures an unsupervised manner, for which we believe our work is one of the first, seems promising even without sparsity or minimality assumptions.
INTRODUCTION AND RELATED WORK
Causal discovery seeks to identify the underlying structure of the data generation process by ex- ploiting an appropriate class of assumptions (Spirtes et al., 1993; Pearl, 2000). Despite its success in certain domains, most existing work either focuses on estimating the causal relations between ob- served variables (Spirtes & Glymour, 1991; Chickering, 2002; Shimizu et al., 2006), starts from the premise that causal variables are given beforehand (Spirtes et al., 2013). Real-world observations (e.g., image pixels, sensor measurements, etc.), however, are not structured into causal variables to begin with. Estimating latent causal variable graphs from observations is particularly challenging as the latent variables, even with independent factors of variation (Locatello al., 2019), are not identifiable or "uniquely" recoverable in the most general case (Hyvärinen & Pajunen, 1999). There exist several pieces of work aiming uncover causally related latent variables. For instance, by exploiting the vanishing Tetrad conditions (Spearman, 1928) one is able to identify latent variables models (Silva et al., 2006), and the so-called Generalized Independent Noise (GIN) condition was proposed to estimate linear, non-Gaussian latent variable causal graph (Xie al., 2020), with follow-up studies such as (Adams et al., 2021). However, these approaches are constrained within linear relations, need certain types of sparsity minimality assumptions, and require a relatively large number measured variables as children of the latent variables. The work of (Bengio al., 2019; Ke et al., 2019) used "quick adaptation" the training criteria for learning latent structure but the identifiability results have not been theoretically established yet.
Recent advances in the theory of nonlinear Independent Component Analysis (ICA) have proven strong identifiability results (Hyvarinen & Morioka, 2016; 2017: Hyvarinen et al., 2019; Khe- makhem et al., 2020; Sorrenson et al., 2020) by exploiting certain side information in addition to independence. By assuming that the generative latent factors 2, are conditionally independent
Equal contribution. Code: httpps:/gggteee comwwel wawee/raaaaaaaa 1eap

---

Published as a conference paper at ICLR 2022
given auxiliary variables u that may be time index, domain index, class label, etc. and augmenting observation data x with u. deep generative models fit with such tuples (x, u) may be identifiable in function space; they can recover independent factors up to certain transformation of the original latent variables under proper assumptions (note that we use "latent factor" and "latent processe" interchangeably). Although the temporal structure is widely used for nonlinear ICA, existing work that establishes identifiability results considers only independent sources, or further with linear tran- sitions. However, these assumptions may severely distort the results if the real latent factors have causal relations in between, or if the relations are nonlinear. It is not clear yet how the temporal structure may help in learning temporally causally-related latent factors, together with their causal structure, from temporal observation data.
In this paper, we focus on the scenario where the observed temporal variables x, do not have direct causal edges but are generated by latent processes confounders 2 that have time-delayed causal relations in between. That is, the observed data x, are unknown nonlinear (but invertible) mixtures of the underlying sources: X, g(z)). Our first goal is hence to understand under what conditions the latent temporally causal processes can be identified. Inspired by real situations, we consider both nonparametric, nonstationary setting and parametric setting for the latent processes. In the nonparametric setting, the generating process of each latent causal factor 2 is characterized by nonparametric assignment zie ((Pa(zzz) (az),, ea), in which the parents of žдt (i.e., the set of latent factors that directly cause zzz) together with noise term ea ~ Pe (where Pe denotes the distribution of Ex) generate 25 via unknown nonparametric function fo with some time delay. In the parametric setting, the time-delayed causal influences among latent factors follow linear form. In both set- tings, we establish the identifiability of the latent factors and their causal influences, rendering them recoverable from the observed data.
(Non-)Stationary Time-Series
Exploit Nonstationarity or Functional Form
Latent Causal Variable Learning
Causal Graph Estimation
Mixing Function
Embedding
Variable
Transition Dynamic
X, = 8(2,)
Model Estimation
X1 XL
Enc
Z 2g.
Dec
X
X
Causal Visuelization
1
pa(2,))
2
Zn = (Pa(.))) (4)
7, Erst B,
Figure 1: Our approach: we leverage nonstationarity in process noise or functional and distribu- tional forms of temporal statisties to identify temporally causal latent processes from observation.
Our second goal is then to develop theoretically- sooreiicallyggroundeed training framework that enforces the assumed conditions through proper constraints. To this end, we propose Latent tEmporally cAusal Processes estimation (LEAP), a novel architecture that extends VAËS with a learned causal process prior network that enforces the Independent Noise (IN) condition and models possible nonstation- arity through flow-based estimators. We evaluate LEAP on a number of synthetic and real-world datasets, including video and motion capture data with the properties required by our conditions. Experimental results demonstrate that temporally causal latent processes are reliably identified from observed variables under different dependency structures, and that our approach considerably out- performs existing methods which do not leverage history or nonstationarity information.
The closest work to ours includes (Klindt et al., 2020; Hyvarinen & Morioka, 2017), which require the underlying sources to be mutually independent for identifiability. Our work extends the theories for the discovery of conditionally independent sources the setting with temporally causally-related latent processes, by leveraging nonstationarity or functional and distributional constraint on temporal relations. To the best of our knowledge, this is one of the first works that successfully recover dime-delayed latent processes from their nonlinear mixtures without using sparsity or minimality assumptions. The proposed framework may serve as an alternative tool for creating versatile models robust to domain shifts and may be extended for more general conditions with changing causal relations over time.
2
IDENTIFIABILITY THEORY
2.1 IDENTIFIABILTYY PROPERTY
We summarize the recent literature related to our work from four perspectives and compare our proposed theory with them in Table 1. The detailed comparisons of the problem settings are given in Appendix A.4. Following prior work, we define identifiability in representation function space.
2

---

Published as a conference paper at ICLR 2022
Table 1: Attributes of existing theories. green check denotes that method has an attribute,
whereas red cross denotes the opposite. indicates an approach we implemented.
Approach
Temporal Data Causall--related Factors
Nonparametric Expression
Stationary Process
(Hyvarincn Morioka,
PCL PCI. (Hyvarinnnnn & Morioka, 2017)
GCL (Hyvarinca al. 2019)
IVAE (Khemakhem
GIN (Sorenson al., 2020)
HM-NLICA (Halivà & Hyvarinen, 2020)
Slow (Klindt et 2020)
VausalVAE (Yang at.,, 2021)
X
(Theorem
LEAP (Theorem 2)
Definition (Componentwise Identifiability) Let x, be sequence of observed variables gener- ated by the true temporally causal latent processes specified by (ft, and nonlinear mixing func- tion 9- given in the introduction. learned generative model (ÿ, fe.Pe.) ) is observationally equiva- lent to (g. Pi.Pe,) if the joint distribution i.. (xx) matches Pa.f-pe (X+) everywhere. We say latent causal processes are identifiable if observational equivalence can always lead to identifiability of the latent variables up permutation π and component-wer invertible transformation T:
Pg.f. (xt) = Pa. (x+) = = Tox.
Once the latent causal processes identifiable up componentwise transformations, latent causal relations are also identifiable because conditional independence relations fully characterize time- delayed causal relations in a dime-delayed causally sufficient system. Note that invertible compo- nentwise transformations on latent causal processes do not change their conditional independence relations.
2.2 OUR PROPOSED CONDITIONS
We consider two novel conditions that ensure the identifiability of temporally causal latent processes using (1) nonstationarity (2) functional and distributional constraints on their temporal relations. The corresponding identifiability the latent processes is established in the following two theorems, with proof's and discussions of the assumed conditions provided in Appendix A.
Theorem (Nonparametric Processes) Assume nonparametric processes in Eq. 2, where the tran- sition functions fü are third-order differentiable functions and mixing function g is injective and differentiable almost everywhere; let Pa(zıt denote the of (time-delayed) parent nodes of zuz
x = x==))
Zet fo (12ja-r (R E Pa(zit) Ex)) with Est ~ Peilu
Nonlincor mixing
Nomparametric transition
Nonstationary noise
Here we assume:
1. (Nonstationary Noise): Noise distribution Peslu is modulated (in any way) by the observed cate- gorical auxiliary variables u, which denotes nonstationary regimes or domain index;
2. (Independent Noise):
The noise terms Ca are mutually independent (iee. spatially and temporally independent) each regime of u (note that this directly implies that t are independent from Pa(zu) in each regime);
3. (Sufficient Variability): For any Zς E RR" there exist 2n + values for u, i.e., uj with
j 0, 1,.,... 2n, such that the 2n vectors w(ze, Uz+1) w(2,, 11)), with 0, 1,.,. 2n, linearly independent with w(z,, u) defined below, where qi is the log density of the conditional distribution and Zns-
{Zz-T} denotes history information up maximum time lag L:
(8qi (8qı(zuc (210 Znc, u)
0qn (Zut Bis, w) 8 Q2qi(e pzqi(z Zn., u)
8gnn2n Bibs u)
w(ze, u)
(3)
0zi
8znt
Ozf.
Ozh
Then the componentwise identifiability property of temporally causal latent processes is ensured.
Theorem (Parametric Processes) Assume the vector autoregressive process in Eq. 4, where the state transition functions are linear and additive and mixing function is injective and differentiable almost everywhere. Let B, E RRax be the state transition matrix at lag т. The process noises eat
3

---

Published as a conference paper at ICLR 2022
Figure LEAP: Encoder (A) and Decoder (D) with MLP or CNN for specific data types; (B) Bidirectional inference net- work that approximates the pos- teriors of latent variables 81:7, and (C) Causal process net- work that (1) models nonsta- tionary latent causal processes 24 with Independent Noise con- straint (Thm 1) or (2) models the linear transition matrix with Laplacian constraints (Thm 2).
X1
XL
Sy
A. Enc.
Enc.
A. Enc.
Causal Process
Element/ Channal Indax
Transition
Modallngg pit, pik---ki P
Le-
Time dolayed Information
Index
Dec.
Dec.
D. Dec.
assumed be stationary and both spatially and temporally independent:
L
x = g(z))
2,
Σ B,Z +
with Eit ~ Pcs
(4)
Nontinear mixing
Independent noise
Linear additive transition
Here assume:
1. (Generalized Laplacian Noise): Process noises Ga. Pes are mutually independent and follow the generalized Laplacian distribution Pe; 21([/aa)
a; exp (~>alra)"*) with < 2;
2. (Nonsingular State Transitions): For at least one т, the state transition matrix B, is full rank.
Then the componentwise identifiability property of temporally causal latent processes is ensured.
LEAP: LATENT TEMPORALLY CAUSAL PROCESSES ESTIMATION
Given our identifiability results, we further propose a Latent tEmporally cAusal Processes (LEAP) estimation framework, which built upon the framework of VAEs while enforcing the conditions in Section 2.2 as constraints for identification of the latent causal processes. The model architecture shown in Fig. 2. Here X1:7 and &1TT are the observed and reconstructed time series, and H. and H, denote the forward and backward embeddings. Implementation details are in Appendix C.
3.1 CAUSAL PROCESS PRIOR NETWORK
We model latent causal processes the learned prior network. To enforce the Independent Noise (IN) condition, latent transition priors p( (P((z)) are reparameterized into factorized noise dis- tributions using the change of variable formula. To enforce the nonstationary noise condition, the noise distributions are learned by flow-based estimators, and independence constraints enforced through the contrastive approach. Finally, for interpretation purposes of the causal relations, we use pruning techniques based masked inputs and soft-thresholding.
3.1.1 TRANSITION PRIOR MODELING
Nonparametric and parametric transition priors are modeled below, leading to two separate methods. The IN condition used to reparameterize transition priors into factorized noise distributions.
Nonparametric Transition We propose novel way inject the IN condition into the transition prior. Let (r,) be set of learned inverse causal transition functions that take the estimated latent causal variables and output the noise terms, i.e., &sx (2u. (2.-r)). We model each output component éμ with a separate Multi-Layer Perceptron (MLP) network, so we can easily disentangle the effects from inputs to outputs. Design transformation A B with low-triangular Jacobian as follows:
(InC
2 R......... 20-11 2, mapped to 22-1..... 2 21-1,
with JA JA-BB
(5)
diag 05.e
B
4

---

Published as a conference paper at ICLR 2022
By applying the change of variables formula to the map from A to B and because of the IN condition in Assumption 2 of Theorem 1, one can obtain the joint distribution of the latent causal variables as:
log p A log B + log (ldet (JA-B)l)
(6)
log (2, (ZI-L..... 21 Σ logr(é//u) log (Idet (JA-B)l).
Because the comdition (See Assumption 2 in Thm
The transition prior p (2rr H=i) can thus be evaluated using factorized noise distributions by cancelling out the marginals of time-delayed causal variables both sides of Eq. 7. Given that this Jacobian is triangular, we can efficiently compute its determinant as II:
dr:
82
Òr:
log p (2
log pogp(élu
Σ log
Parametric Transition A group of state transition matrices fr-} is used model the inverse transition functions: & = 2; - E.= r-22--- Because additive noise, the transition priors can be directly written in terms of factorized noise distributions.
logp (2.1(2)-
> logg((e)
3.1.2 NONSTATIONARY NOISE ESTIMATION
A flow-based density estimator is used fit the residuals (e.g., non-Gaussian noises) and score the likelihood in Eqs. 8 and 9. The independence of residuals is enforced inside the density estimator.
Flow-based Noise Estimation We apply the componentwise neural spline flow model (Dolatabadi et al., 2020) to fit the estimated noise terms. The distribution of each noise component 6xt Pejlu modeled separately by transforming standard normal noises through linear rational splines 85,-- To model nonstationarity, we keep copy of spline flows {St,u) for each nonstationary regime
ds, (Gs)
and trigger it when data falls into this category: p(ε = PN((.))) (s))
Stationary
de,
Nonstationary noise condition
sources are considered as special cases where the nonstationary regime u = 1 for all data samples.
Independence by Contrastive Learning We further force the estimated noises (é to be mutu- ally independent (corresponding the IN condition) across values of (i, Similar to FactorVAE (Kim & Mnih, 2018), we train a discriminator D({Er)) together with the latent variable model, to distinguish positive samples which are the estimated noise terms, against negative samples {2 which are random permutations of the noise terms across the batch for each noise dimension (i. t). The Total Correlation (TC) of noise terms can be estimated by the density-ratio trick and is added to the Evidence Lower BOund (ELBO) objective function to enforce joint independence of noise 2((aa)) terms: LTC ((aa)>(g(aa) log I-D((e.))
3.1.3 STRUCTURE ESTIMATION
For visualization purposes, we apply sparsity-encourageee regularizations to the learned causal re- lations in latent processes. Specifically, we use a combination of masked inputs and pruning ap- proaches. Note that our identifiability results do not rely sparsity of causal relations in latent processes. used only for visualizing causal relations when the causal processes are nonlinear.
Masked Input and Regularization For latent processes of sparse causal relations, each MLP of the inverse transition function r; has a learned n-dimensional soft mask vector a(x) E [0, 1] with the j-th time-delayed inputs of the ML.P being multiplied by ( A fixed Ta penalty is added to the mask during training: L = I|a [[t)))) For linear transition, the penalty is added to the transition matrices since the weights directly indicate whether an edge exists.
Pruning For nonlinear relations,
LassoNet (Lemhadri et al., 2021) post-processing
step to remove weak edges. This approach prunes input nodes by jointly passing the residual layer and the first hidden layer through hierarchical toft-thresoolding optimizer. We fit the model
5

---

Published as a conference paper at ICLR 2022
subset of the recovered latent causal variables. Though the true causal relations may not follow the causal additive assumption, the pruning step usually produces sparse causal relations.
3.2 INFERENCE NETWORK
A bidirectional Gated Recurrent Unit is used to infer latent variables. We approximate the posterior (0(21:7 X1:1) with an isotropic Gaussian with mean and variance terms from the inference net- work. The KL divergence is LKLD = DEL (9s [l and is estimated via sampling approach because prior distribution is not specified but rather learned by causal process network.
3.3 ENCODER AND DECODER
The reconstruction likelihood is LRcann = Proconstruct (x,) 2,). where Preconstruct is the decoder dis- tribution. For synthetic and point cloud data, decoder and MSE loss used for reconstruction. For video datasets with single objects (e.g.. KiT- TiMask), vanilla CNNs and binary cross-entropy loss are used. For videos with multiple objects, we apply a disentangled design (Kulkarni et al., 2019) with two separate CNNs, one for extracting visual features and the other for locating object locations with spatial softmax units. The decoder re- trieves object features using object locations and reconstructs the scene with MSE loss. The network architecture details are given Appendix C.I.
MLP with LeakyReI.U units encoder and
3.4 OPTIMIZATION
We train the VAE and noise discriminator jointly. The VAB parameters are updated using the aug- mented ELBO objective LELBO. The discriminator is trained to distinguish between residuals from (((x)) and (((()t with Lp. thus learning to approximate the density ratio for estimating Lre:
CELBO 1 EXEN PRccnn 3CXLD - Maas oLre, Lo = tोe ELeN [log - Exe log (1 [[]()))))))
The discussions of the hyperparameter selection and sensitivity analysis are in Appendix C.2.1.
4 EXPERIMENTS
We comparatively evaluate LEAP on a number of temporal datasets with the required assumptions satisfied or violated. We aim to answer the following questions:
Does LEAP reliably learn temporally-causal latent processes from scratch under the proposed conditions? What is the contribution of each module in the architecture?
2. Is bistory/noossationaayy information necessary for the identifiability of latent causal variables?
How do common assumptions in nonlinear ICA (i.e., independent sources or linear relations assumptions) distort identifiability if there are time-delayed causal relations between the latent factors, or the latent processes are nonlinearly related?
4. Does LEAP generalize when some critical assumptions in the proposed conditions are violated? For instance, how does it perform in the presence of instantaneous or changing causal influences?
Evaluation Metrics To measure the identifiability of latent causal variables, we compute Mean Correlation Coefficient (MCC) on the validation dataset, standard metric in the ICA literature for continuous variables. MCC reaches 1 when latent variables are perfectly identifiable up to permutation and componentwise invertible transformation in the noiseless case (we use Pearson correlation and rank correlation for linearly and nonlinearly related latent processes, respectively). To evaluate the recovery performance on causal relations, we use different approaches for (1) linear and (2) nonlinear transitions: (1) the entries of estimated state transition matrices are compared with the true ones after permutation, signs, and scaling are adjusted, and (2) the estimated causal skeleton is compared with the true data structure, and Structural Hamming Distance (SHD) is computed.
Baselines and Ablation We experimented with three kinds of nonlinear ICA basclines: (1) Be- ta VAE (Higgins et al., 2016) and VAE (Kim Mnih, 2018) which ignore both history and nonstationarity information; (2) iVAE (Khemakhem et al., 2020) and TCL. (Hyvarinen & Morioka, 2016) which exploit nonstationarity to establish identifiability, and (3) Slow (Klindt et al., 2020) and PCL (Hyvarinen & Morioka, 2017) which exploit temporal constraints but assume independent sources. Model variants are built to disentangle the contributions of different modules. As in Table 2, we start with VAE and add our proposed modules successively without any change on the training settings. Finally, (4), we fit our LEAP that uses linear transitions (LEAP- VAR) with nonstationary data to show if linear relation assumptions distort identifiability.

---

Published as a conference paper at ICLR 2022
4.1 SYNTHETIC EXPERIMENTS
We first design synthetic datasets with the properties required by NonParametric (NP) and para- metric Vector AutoRegressive (VAR) conditions Section 2.2. We set latent size n 8. The lag number of the process is set to L 2. The mixing function g is random three-layer MLP with LeakyReLU units. We give the data generation procedures for nonparametric conditions, parametric conditions, and five types of datasets that violate our assumptions separately in Appendix B.1.
MCC-0.971
Estimated Skeleton
Skeleton
NP Processes
0:08
12
LEAPVAR
0:17
BetaVAEE
Thae Iatortes
-0.05
0.99
0.17
0.12
0.17
INAE
00.11
0.00
07
0.20
0.31
0.07
0.94
0.20
03254667
MCC
TCL
(b)
-0.05 0.08
24
0.04
0.98
0.25
0.43
0.13
SloWAE
-0.06
0.05
099
0:12
0.31
0.27
0:43
0.06
0.07
9:20
02
0K
23K
50K
7SK
200K
Estimated
Esimated (c)
Steps
(e)
(d)
Figure 3: Results for synthetic nonparametric processes (NP) datasets: (a) MCC for causally-related factors; (b) recovered causal skeletons with (SHD=5); (c) scatterplots between estimated and true factors; and (d) MCC trajectories comparisons between LEAP and baselines.
Main Results Fig. 3 gives the results on NP datasets. The latent processes are successfully re- covered, indicated by (a) high MCC for the casually-related factors and (b) the recovery of the causal relations (SHD=5). Panel (c) suggests that the latent causal variables are estimated up to per- mutation and componentwise invertible transformation. The comparisons with baselines are in (d). In general, the baselines that do not exploit history or nonstationarity cannot recover the latent pro- cesses. Slow VAE and PCL distort the results due to independent source assumptions. Interestingly, LEAP-VAR, which uses linear causal transitions, gains partial identifiability NP datasets. This might promote the usage of linear components of transition signals to guide the learning of latent causal variables.
The results for VAR datasets are in Fig. 4. Sim- ilarly, the latent processes are recovered, as indi- cated by (a) high MCC and (b) recovery of state transition matrices. The latent causal variables are estimated up to permutation and scaling (c). The
Table 2: Contribution of each module to MCC.
Module
NP (Dense)
VAR
Buseline (3-VAE)
0.446 1: 0.004
0.495 +:2 0.007
Causal Process Prior
0.721 + 0.121
0.752 = 0.035
baselines without using history or assume indepen-
Nonstationary Flow
0.939 0.008
0.935 = 0.014
dent sources again fail to recover the latent pro-
Noise Discriminator
0.983 1 0.002
0.978 1 0.004
cesses (d). We show the contributions of different components of LEAP in Table 2. Causal process prior and nonstationary flow significantly improve identifiability. Noise discriminator further increases MCC and reduces variance. Note that we use dense network without the input masks on the NP dataset for ablation studies, to validate that our proposed framework does not rely on sparse causal structure for latent causal discovery.
Robustness We show the consequences of the violation of each of the assumptions on the synthetic datasets to mimic real situations. For VAR processes, we create datasets: (1) with causal relations changing over regime, and (2) with instantaneous causal relations, (3) with Gaussian noise, and (4) with low-rank state transition matrices. For NP processes, we violate (5) the sufficient variability by creating datasets with fewer than the required 2n + 1 = 17 regimes. We fit LEAP on these datasets without any modification. Our framework can gain partial identifiability under (1,4) but violating (2) conditional independence and (3) Con-Gaussiantty distort the results obviously. Our approach, although designed to model nonstationarity by noise, can be extended to model changing causal relations. The partial identifiability of (4) is because the low-dimensional projections of the latent processes are recovered, as illustrated by a numerical example in Appendix A.2.4. For NP processes, nonstationarity is necessary for identifiability. Furthermore, the differences of the MCC trajectories under 15 and 20 regimes seem marginal, suggesting that our approach does not always require least 2n + 17 regimes to achieve full identifiability of the latent processes.

---

Published as a conference paper at ICLR 2022
MCC-0 973
Entries of
Entriss B,
VAR Processes
0:05
wogir
LEAP
045
BelaVAE
-5
PCL
SloWVAE
Estimates
(b)
0.4
0.97
35K
534
T0K
100K
Estimated
Estimated
Steps
(a)
(c)
(d)
Figure 4: Results for synthetic parametric processes (VAR) datasets: (a) MCC for causally-related factors; (b) scatterplots of the entries of B,; (c) scatterplots between estimated and true factors; and (4) MCC trajectories comparisons between LEAP and baselines.
Violations
NP Processes w/ Violations
1.00
1,00
6.75
relations
0.75
Statimary onary regineses
MCC
0.50
Instantaneous rolations
0.30
6.25
Low-rank transitions
Gaussian naise
0.25
honston
0.50
0.00
Steps
Figure 5: MCC trajectories of LEAP for temporal data with clear assumption violations.
4.2 REAL-WORLD APPLICATIONS: PERCEPTUAL CAUSALITY
Three public datasets including KiTTiMask (Klindt et al., 2020), Mass Mars-Spring system (Li et al., 2020), and CMU MoCap database are used. The data descriptions are in Appendix B.2; depending on its property (e.g., whether it has multiple regimes), we apply the corresponding method. We first compare the MCC performances between our approach and the baselines on KiTTiMask and Mass-Spring system in Fig. 6. Our parametric method considerably outperforms the baselines that do not use history information. Because the true latent variables for CMU-MoCap are unknown, we visualize the latent traversals and the recovered skeletons in Appendix D.I, qualitatively comparing our nonparametric method with the baselines in terms of how intuitively sensible the recovered processes and skeletons are.
KITTiMask (ind Sources)
Spring System (VAR)
LEAP
0.8
MCC
SlowVAE
0.6
BetaVAE
0.4
FactorVAE
PCL
0.4
02
0K
10K
15K
20k
2K
10K
Steps
Steps
Figure 6: MCC trajectories comparisons on KiTTiMasks and Mass-Spring system.
Parametric Transition KiTTiMask LEAP with VAR transitions is used. We set latent size n 10 and the lag number
1. The gap between our result and that by Slow relatively small, seen in Fig. 6; this is because the latent processes on this dataset seem rather independent (according to the transition matrix learned by LEAP, given in Fig. 7(c)) and when the latent processes are independent, our VAR method reduces to Slow as its special case. As shown in Fig. 7, the latent causal processes are recovered, as seen from (a) high MCC for independent sources; (b) latent factors estimated up to componentwise transformation; (c) the estimated state transition matrix, which is almost diagonal (independent sources); and (d) latent traversals confirming that the three latent causal variables correspond to the vertical, horizontal, and scale of pedestrian masks.
Parametric Transition Mass- System Mass-Spring system is a linear dynamical system with ball locations (zt, yf) as state variables and lag number = 2. LEAP with VAR transitions is used. In Fig, 8, the time-delayed cross causal relations recovered (a) causal variables keypoint (x/, v are successfully estimated; (b) the spring connections between balls are recovered (SHD=0). Visualizations of recovery of latent variables and skeletons showcased in Appendix D.2.
8

---

Published as a conference paper at ICLR 2022
MCC=0.908
Entries B1
True Latents
0.99
0.04
0.02
Vertell
Cause index
0.93
0.03
0.00
Postion
0.86
Horizontad
0.000
0.90 0.01
0.00
0.01
0.00
Estimated
1
Estimated Latents
Effect Index
(b)
(c)
(d)
Figure 7: KiTTiMask dataset results: (a) MCC for independent sources; (b) scatterplots between estimated and factors; (c) entries of B and (d) latent traversal on fixed video frame.
2:52
5:43
0.00
0.00
3.03
0.00
0:50 0.00 0
0:00
9.02
0.00
18.00
0:00
0.50 0.00
0:05
0:50
0.00
2:00
0:00
0L50
9.02
5.06
0.E0
91
8.00
0:62
0.00 0.00
1:32
8.40
5.00
10.00
00
6:00
6:80 0.00 0.00
0:00
0.00
4.40
0.00
0.00
0.00
0.00
0:62
0.05
6:93
1:C0
0:33
6:52
0.00
0.02
0.00
0.00
0.00
3.00
8.00
0.00
0.00
1:35 4440
0:00
0.00
0.00
0:02
0.00 0:00
0.00
0.02
00000.00
0:2
0-02
0.13
3.50
0.06
9.02
0.20
0.03
$.00
0.60
0.00
0.00
2.02
8:02 0.00 0.00
0.00
5.03
5:00 0.00
1:37 /0
0:00
0.00
0.00
0.02
0.03
0,00
0.0.3
0.02
6.09
5.00
0.00 3000
8:50
0.00
0.00 0.00
0:60
0.00
0102
6.00
0.004.60
0.50
0.03 9.00
0-0.0
0.30
0:43
8:00
5.00
0.60
0.00
0.55
0.00
xi- Ni--5--13 Xi-s, Ni-s Si-yy-- Ni-g.
197
Ball
Ball
Ball
Ball
Ball
(a)
(b)
Figure 8: Mass-Spring system results: (a) MCC for causally-related sources; (b) entries of B
Nonparametric Transition CMU--MCCa We fit LEAP with nonparametric transitions 12 trials of motion capture data for subject with 62 observed variables of skeleton-based measure- ments each time step. The 12 trials contain walk cycles with slightly different dynamics (e.g.. walk, slow walk, brisk walk). We set latent size n = 8 and lag number L = 2. The differences between trials are modeled by nonstationary noise with one regime for each trial. The results are in Fig. 9. Three latent variables (which seem to be pitch, yaw, roll rotations. respectively) are found to explain most of the variances human walk cycles (Panel c). The learned latent coordinates show smooth cyclic patterns with slight differences among trials (Panel a). Finally, we find that pitch (e.g., limb movement) and roll (c.g., shoulder movement) of human walking are coupled while yaw has independent dynamics (Panel
Estimated Ske eton
Pitch z
Rotation
Yaw 2
Roll x²
72-2
(a)
(b)
(6)
Figure 9: MoCap dataset results: (a) latent coordinates dynamics for 12 trials; (b) estimated skeleton; and (c) latent traversal by rendering the reconstructed point clouds into the video frame.
CONCLUSION AND FUTURE WORK
In this work, we proposed two provable conditions under which temporally causal latent processes can be identified from their observed nonlinear mixtures. The theories have been validated on number of datasets with the properties required by the conditions. The main limitations of this work lie in our two major assumptions: (1) there is no instantaneous causal influence between latent causal processes, and (2) causal influences do not change across regimes. Both of them may not be true for some specific types of temporal data. The existence of instantaneous relations distorts identifiability results, but the amount of these relations can be controlled by time resolution. While do not establish theories under changing causal relations, have demonstrated through exper- iments the possibilities of generalizing our identifiability results to changing dynamics. Extending identifiability theories and framework to accommodate such properties is future directions.

---

Published as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
Our code for the proposed framework and experiments can be found at tatttps:/gitiii com/ weir weirayao/ / leap. For theoretical results, the assumptions and complete proof of the claims are in Appendix A. For synthetic experiments, the data generation process is described in Appendix B.1. The implementation details of our framework are given in Appendix C.
ACKNOWIFDGEMENT
KZ would like to acknowledge the support by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, and by the United States Air Force under Contract No. FA8650-17-C7715. YS and CS would like to acknowledge the support by the National Key R&D Program of China (No. 2018AAA0104000) National Nat- ural Science Foundation of China (No. 61921004), and the Natural Science Foundation of Jiangsu Province of China (No. BK20202006).
REFERENCES
Adams, N. R. Hansen, and K. Zhang. Identification of partially observed linear causal models: Graphical conditions for the non non-gaussian and heterogeneous cases. In Conference Neural Information Processing Systems (NeurIPS), 2021.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: /atv:/901/ 1901. 9901. /0912, 2019.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554 2002.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiiv preprint arXiv: 2014.
D. Danks and S. Plis. Learning causal structure from undersampled time series. In JMLR: Workshop and Conference Proceedings, pp- 1-10, 2013.
Martijn de Jongh and Marek J Druzdzel. A comparison of structural distance measures for causal bayesian network models. Recent Advances in Intelligent Information Systems, Challenging Prob- lems of Science, Computer Science series, pp. 443 456, 2009.
Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Invertible generative modeling using linear rational splines. International Conference on Artificial Intelligence and Statistics, pp. 4236-4246. PMLR, 2020.
M. Gong*, K. Zhang* D. Tao, P. Geiger, and B. Schölkopf. Discovering temporal causal relations from subsampled data. In Proc. 32th International Conference on Machine Learning (ICMI. 2015), 2015.
M. Gong, K. Zhang, B. Schölkopf, C. Glymour, and D. Tao. Causal discovery from temporally aggregated time series. In Proc. Conference on Uncertainty in Artificial Intelligence (UAI'17), 2017.
Clive WJ Granger. Implications of aggregation with common factors. Econometric Theory, 3(02): 208-222, 1987.
Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pp. 939-948. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2022
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv: 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. International Conference on Artificial Intel- ligence and Statistics, pp. 2207-2217. PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2658. PMLR, 2018
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007. 20.0000077...... 10930, 2020.
Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. arXiv preprint 1906: 2019.
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, 22(127): 2021.
Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. arXiv preprint arXiv:2007.00631 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114-4124 PMLR, 2019.
Kiyotoshi Matsuoka, Masahiro Ohoya, and Mitsuru Kawamoto. neural net for blind separation of nonstationary signals. Neural networks, 8(3):411-419 1995.
Stanistaw Mazur and Stanislaw Ulam. Sur les transformations isométriques d' espaces vectoriels normés. CR Acad. Sci. Paris, 194(966-988):116, 1932.
Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
Judea Pearl al. Models, reasoning and inference. Cambridge, UK: CombridgeUnivveritsprress. 19, 2000.
Hans Reichenbach. The direction of time, volume 65. Univ of California Press, 1956.
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
11

---

Published as a conference paper at ICLR 2022
R. Silva, R. Scheines, C. Glymour, and P. Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7:191-246, 2006.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-flow networks (gin). arXiv preprint 20t::20000088... 2020.
Charles Spearman. Pearson's contribution to the theory of two factors. British Journal of Psychol- ogy, 19:95-101, 1928.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. Spring- Verlag Lectures in Statistics, 1993.
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 9(1):62-72, 1991.
Peter L Spirtes, Christopher Meek, and Thomas S Richardson. Causal inference the presence of latent variables and selection bias. arXiv preprint arXiv:/302.4985 2013.
Henning Sprekeler, Tiziano Zito, and Laurenz Wiskott. An extension of slow feature analysis for nonlinear blind source separation. The Journal of Machine Learning Research, 15(1)9921-947, 2014.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. General- ized independent noise condition for estimating latent variable causal graphs. arXiv preprint 2XXv:2011000411., 2020.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593-9602, 2021.
Kun Zhang and Aapo Hyvärinen. general linear non non- gaussian state-space model: Identifiability, identification, and applications. In Asian Conference Machine Learning, pp. 113-128. PMLR, 2011.
12

---

Published as a conference paper at ICLR 2022
Supplement to "Learning Temporally Causal Latent Processes from General Temporal Data"
The supplementary materials are divided into five main sections. Appendix A, we provide the explanations of each assumption and give the proof of the identifiability theory. We also give side- by-side comparisons, by providing the mathematical formulations of the closest works and making comparisons in terms of problem setups and critical assumptions. Finally, how the theory is nected to the training framework is discussed. In Appendix B, we provide the details of the synthetic and real-world datasets and explain the evaluation metrics. In Appendix we describe our network architecture, hyperparameters setting, and training details. The additional experiment results are given Appendix D. The related work is summarized Appendix
A Identifiability Theory
14
A.1
Notation and Terminology
14
A.2
Discussion of our Assumed Conditions
14
A.2.1
Independent Noise (IN) Condition
14
A.2.2
Nonstationary Noise and Sufficient Variability Condition
15
A.2.3
Generalized Laplacian Noise Condition
16
A.2.4
Nonsingular State Transitions Condition
16
A.3
Proof of Identifiability Theory
17
A.3.1
Preliminaries
17
A.3.2
Proof of Theorem
18
A.3.3
Proof of Theorem 2
20
A.4
Comparisons with Existing Theories
22
A.5
Connecting Theories to Model
23
Experiment Settings
23
B.1 Synthetic Dataset
23
B.2 Real-world Dataset
24
B.3 Evaluation Metrics
25
Implementation Details
25
C.I Network Architecture
25
C.2 Hyperparameters and Training Details
27
C.2.1 Hyperparameter Selection and Sensitivity Analysis
27
C.2.2 Training
D Additional Experiment Results
29
D.1 Comparisons between LEAP and Baselines on CMU-Mocap Dataset
.attsst.........................
29
D.2 Mass-Spring Systems
30
Extended Related Work
13

---

Published as a conference paper at ICLR 2022
A
IDENTIFIABILITY THEORY
A.1 NOTATION AND TERMINOLOGY
We summarize the notations used throughout the paper in Table A.1.
Table A.1: List of notations.
Index
Time index
i.j
Variable element (channel) index
Time lag index
perm
Random permutated variable index across the data batch
Variable
Xr
Observation data
X,
Reconstructed observation Auxiliary nonstationary regime variable Underlying sources
Z
Time-delayed latent causal variables
Pa(aa)
Set of direct cause nodes/parents of node Zit Measurement error
H. F
Forward backward embeddings in bidirectional RNN Soft mask vector
(+)
w
Modulation parameter vector
B
State transition matrix
Cat
Process noise term Estimated process noise term Estimated sources
Za
True underlying sources
Function and Hyperparameter
p
Distribution function (c.g., Pe., is the distribution a..)
9
Arbitrary nonlinear and injective mixing function Nonlinear transition function for 2,t
fr
Indeterminancy mappings between z, and 2; Learned inverse transition function for residual é Spline flow function for residual & in regime u Weights in the augmented ELBO objective Latent size
8
B,Y,
L
Maximum time lag Total length of time series
Permutation operation
Component-we wise invertible nonlinearities
q
Log density function Parameters Laplacian distribution
A.2 DISCUSSION OF OUR ASSUMED CONDITIONS
We first explain and justify each critical assumption in the proposed conditions. We then discuss how restrictive mild the conditions are real applications.
A.2.1 INDEPENDENT NOISE (IN) CONDITION
The IN condition was introduced in the Structural Equation Model (SEM), which represents effect as a function direct causes X and noise E:
f(X with
X XL
(10)
IN condition
14

---

Published as a conference paper at ICLR 2022
If X and Y do not have common cause, seen from the causal sufficiency assumption of structural equation models in Chapter 1.4.1 of Pearl's book (Pearl et al., 2000), the IN condition states that the unexplained noise variable E' is statistically independent of cause X. IN is a direct result of assuming causal sufficiency in SEM. The main idea for the proof is that if IN is violated, then by the common cause principle (Reichenbach, 1956), there exist hidden confounders that cause their dependence, thus violating the causal sufficiency assumption. Furthermore, for a causally sufficient system with acyclic causal relations, the noise terms in different variables are mutually independent. The main idea is that when the noise terms are dependent, it is customary to encode such dependencies by augmenting the graph with hidden confounder variables (Pearl et al., 2000), which means that the system is not causally sufficient.
In this paper, we assume the underlying latent processes form casually-sufficien system without latent causal confounders. Then, the process noise terms are mutually independent, and more- over, the process noise terms Gtt are independent of direct cause/parent nodes Pa(zçe because of time information (the causal graph is acyclic because of the temporal precedence constraint).
Applicability Loosely speaking, there are no latent causal confounders in the (latent) causal processes and the sampling frequency is high enough observe the underlying dynamics, then the IN condition assumed in this paper is satisfied in a causally-sufficien system and, moreover, there is no instantaneous causal influence (because of the high enough resolution). At the same time, we acknowledge that there exist situations where the resolution is low and there appears to be instan- taneous dependence. However, there are several pieces of work dealing with causal discovery from measured time series in such situations; see, e.g., Granger (1987); Gong et al. (2015); Danks & Plis (2013); Gong et al. (2017). In case there are instantaneous causal relations among latent causal pro- cesses, one would need additional sparsity or minimality conditions to recover the latent processes and their relations, as demonstrated Silva et al. (2006); Adams et al. (2021). How to address the issue of instantaneous dependency or instantaneous causal relations in the latent processes will be one line of our future work.
A.2.2 NONSTATIONARY NOISE AND SUFFICIENT VARIABILITY CONDITION
Nonstationary Noise For nonparametric processes, temporal constraints are not sufficient for the identification of latent causal transition dynamics whose functional or distributional form is not con- strained. Otherwise, there is no need for Theorem 2 to assume the generalized Laplacian noise and the full-rankness of state transitions at all. In this paper, an alternative way is exploit the (tem- poral) nonstationarity of the data caused by changing noise distribution (hence called nonstationary noise condition). We assume the functions of the temporal causal influences denoted by fi remain the same across across the |u| regimes or domains of data we have observed, but the distributions Pe,lu: of noise terms that serve as arguments to the structural equation models, may change. One special case of this principle uses nonstationary variances, i.e., the noise variances change across nonstationary regimes. This kind of perturbation has been widely used in linear ICA (Matsuoka et al., 1995). Additionally, the nonstationary noise condition in this paper allows for any kinds of modulation of noise distribution by nonstationary regimes u, such as changing distributional forms, scale, and location by u, as long the modulated sources satisfy sufficient variability condition described below.
Sufficient Variability The sufficient variability condition was introduced in GCL (Hyvarinen et al., 2019) to extend the modulated exponential families (Hyvarinen & Morioka, 2016) to gen- eral modulated distributions. Essentially, the condition says that the nonstationary regimes u must have a sufficiently complex and diverse effect on the transition distributions. In other words, if the underlying distributions are composed of relatively many domains of data, the condition generally holds true. For instance, in the linear Auto-Regressive (AR) model with Gaussian innovations where only the noise variance changes, the condition reduces to the statement in (Matsuoka et al., 1995) that the variance of each noise term fluctuates somewhat independently of each other in different nonstationary regimes. Then the condition is easily attained if the variance vector of noise terms in any regime is not a linear combination of variance vectors of noise terms in other regimes.
We further illustrate the condition using the example of modulated conditional exponential families in (Hyvarinen et al., 2019). Let the log-pdf q(2, {ze-r}, u) be conditional exponential family
15

---

Published as a conference paper at ICLR 2022
distribution of order l given nonstationary regime and history ZHz {z1-r}:
((zu) BHK, u) (z((zt)) + Σ Vig (a(t (Z u) log Z(zHK, u),
(11)
where % is the base measure, Qay the function of the sufficient statistic, Xij is the natural parameter, and log Z is the log-partition. Loosely speaking, the sufficient variability holds if the modulation of by u on the conditional distribution Za ZHkx u) is not too simple in the following sense:
1. Higher order of k (k > 1) is required. If = 1, the sufficient variability cannot hold;
2. The modulation impacts As by u must be linearly independent across regimes u. The sufficient statistics functions qú cannot be all linear, i.c., we require higher-order statistics.
Further details of this example can be found in Appendix B of (Hyvarinen al., 2019). In summary, we need the modulation by u to have diverse (i.e., distinct influences) and complex impacts on the underlying data generation process.
Applicability The nonstationarity of process noise seems to be prominent in many kinds of tem- poral data. For example, nonstationary variances are seen in EEG/MEG, natural video, and closely related to changes in volatility in financial time series (Hyvarinen & Morioka, 2016). As assume the transition functions fi are fixed across regimes, the data that most likely satisfy the proposed con- dition is collection of multiple trials/segments of data with slightly different temporal dynamics in between, where the differences can be well modeled by different noise distributions. For instance, in MEG data, temporal nonstationarity can be modeled by segmenting the measured data into different sessions (e.g., stimuli, rest, etc.) where the session index modulates the noise variance.
A.2.3 GENERALIZED LAPLACIAN NOISE CONDITION
In the parametric (VAR) processes in Theorem tions to achieve identifiability. Specifically, we constrain the process noise distribution to be within the generalized Laplacian distribution family in this paper. This L1-sparse temporal prior is moti- vated by the natural statistics video data, where the uncertainty could have sharp impacts on some latent factors, but most other factors are not perturbed between two adjacent frames. This transi- tion prior has strong connections with slow feature analysis (Sprekeler et al., 2014; Klindt et al., 2020) which measures slowness in terms of the L2 distance between temporally adjacent encod- ings as temporal constraints for nonlinear ICA. Note that although the Laplacian-lin like distributional form is pre-defined, the generalized Laplacian distrbution can still be used to broad family of perturbations with different shapes by changing and \ of the distribution.
exploit the non non-Gaussianity of noise perturba-
Applicability LI-sparse transition priors are widely used to model video datasets and natural scene measurements. This condition is applicable to video datasets where the external factors have sharp effects on some but not all latent factors two adjacent frames.
A.2.4 NONSINGULAR STATE TRANSITIONS CONDITION
Nonsingularity is standard assumption made in the previous studies (Zhang & Hyvärinen, 2011) to achieve identifiability of linear state-space models. We give a two-dimensional low-rank vector autoregressive (VAR) process example below to illustrate the concept. Define low-rank VAR process with time lag L = 1 below:
1
2, BZ- + with
[a 6],
(12)
axa axb
where 2, = [21ee 22e] and E, = [Eies 62c]
Multiplying both sides of the VAR process with a row
vector (a, b). we have:
16

---

Published as a conference paper at ICLR 2022
aa
|a
bJ
x
2z + la b) x Et
(13)
[
21
(a + ba) la
b) x Z + (a. b) x c
(14)
Hence, the two-dimensional VAR process reduces to single linear AR process with 2, = la b) x
24 0z14 + baz and é, = [a x E, = 16 + be2t, which is linear combination the original two processes. In this case, we cannot recover 2, at all, but only the linear combination 2,-
In summary, when the state transition matrices are not of full rank, there exist low-dimensional projections of the underlying latent processes that satisfy the observational equivalence everywhere. By assuming the nonsingular state transitions, one could prevent from recovering low-dimensional projections of latent causal factors and time-delayed relations.
A.3 PROOF OF IDENTIFIABILITY THEORY
A.3.1 PRELIMINARIES
Equivalent Relations on Latent Space Our proof of identifiability starts from deriving relations on estimated latent space from observational equivalence: the joint distribution Pa.f.p. (XHx Xt) matches Py.f.p. (XHx, Xx) everywhere. Note that we consider only one future time step X for sim- plicity the joint probability of the whole sequence can be decomposed into product of these terms. Since the learned mixing function X; = g(z)) can be written as = [g (g) gz) because of injective properties
(g.9), we can see that I = g o ((g) o g) h = (g) I on the latent space. Our goal here really to show that this function h, which represents the indeterminancy of the learned latent space, is permutation with component-wise nonlincarities. It has been proved in (Klindt et al., 2020) that:
h for some function
1. Indeterminancy h on latent space only be bijection on the latent space if both g and are injective functions, and h preserves the prior distribution in the latent space. The proofs are in Appendix A.1 of (Klindt et al., 2020) Page 18;
2. Eq. 15 can be directly derived from observational equivalence using the injective properties of (g. g). The proofs are Appendix A.1 of (Klindt et al., 2020) on Page 19:
p(z+)
p(zi)(z(-+)) = p(h p(h--'zz)ffb (z))((h"(<x++)))
p(h p(h))(2))) Y(zi: (z(-r))
(15)
Identifiability of Linear Non-Gaussian State-Space Model Linear State Space Model (SSM) defined below has been proved to be fully identifiable in (Zhang & Hyvärinen, 2011) when both the process noise e and measurement error e₁ are temporally white and independent of each other, and at most one component of the process noise & is Gaussian. The observation error e can be either Gaussian non-Gaussian:
Azi + et:
(16)
2, Σ B B Ex.
(17)
We will make use of this property of linear non-Gaussian SSM for deriving Theorem 2. The main
idea is if we can prove h of parametric conditions (which also has vector autoregressive processes in Eq. 17 with non-Gaussian noise) is within affine transformations, the componentwise identifi- ability of true latent variables can be directly derived because we can treat the affine indeterminacy "high-level" affine mixing of sources same A in Eq. 16 without measurement error.
17

---

Published as a conference paper at ICLR 2022
A.3.2 PROOF OF THEOREM 1
Theorem A.1 (Nonparametric Processes) Assume nonparametric processes in Eq. 18, where the transition functions fü are third-order differentiable functions and mixing function is injective and differentiable almost everywhere; let Pa(zz) denote the set of (time- delayed) parent nodes of 2ix
x g(
Zit
(133 Pa(zu)),
with
Cat ~ Peilu
(18)
Nonlinear mixing
Nonparametric transition
Nonstationary noise
Here assume:
1. (Nonstationary Noise): Noise distribution Pe/lu is modulated (in any way) by the observed cate- gorical auxiliary variables u, which denotes nonstationary regimes or domain index;
2. (Independent Noise):
The noise terms en mutually independent (i.e., sputially and temporally independent) in each regime of u (note that this directly implies that egt are independens from Pa(zu) in each regime):
3. (Sufficient Variability): For any 24 E IR" there exist 2n
values for u, i.e,, U with j 1,,.... ...y 2n, such that the 2n vectors w(ze, Uj+1) w(z,, uz), with 0, 1,.,, 2n, are linearly independent with w w(z:; u) defined below, where qi is the log density of the conditional distribution and R.ts-
((x-r) denotes history information up maximum time lag L;
(8q (211//mss u) Oqn(Ene Bat. u) & Cu(eeeZ
8 8qn Ln Bettes u)
w w(2,, u)
(19)
Ozu
0znt
dzie
dz
Then the componentwise identifiability property of temporally causal latent processes ensured.
Proof: We first extend Eq. 15 to include conditioning on the nonstationary regime u. We then show that if sufficient variability condition is satisfied, the indeterminancy function h only be permutation with component-wist nonlincarities.
Step 1 We first derive equivalent relations on the latent space by conditioning on the nonstationary regime This be directly achieved by applying the change of variable formula on the L invertible maps: 2, = h-1 [zz), 24-1 = h-1 (24-1,,) .... 2t-1. = h=-((x1)). W.l.o.g, let's assume L, == for now. We then have the following three equalities:
Oh (2t)
Oh 0h-
p(z,, 2t-1 1- un p(,- h- (14
det.
det
(20)
Oz:
δz,-1
Oh - (zxe
p(zi)= = (p(h)((z)) dot
(21)
Oh -1 (Z(-1)
p(2- = (( u)
det
(22)
02--1
Solving for the determinant terms in Eq. 21 and Eq. 22 and plugging them into Eq. 20, we have:
p(zz)
p (z, u) = (1-1 - (Z:-1), u)
(23)
p(h (2())
It is straightforward to see that this relation holds for multiple time lags where L > We take logs on both sides, and we define ((2,) & q(zz) the marginal log-density of the components 2, when u is integrated out. We then have:
( u) a(h (olh-((z))l
((2,7), (x0))_)---"(((),,,,))_ = -(())= 7(1 (())),
(zx))l
(24)
and using the Independent Noise (IN) assumption. the conditional log-pdf q(z, ( u) and its estimated version (h 1 (z, [/h ()) u) are conditional independent (note this has been en- forced in causal process network constraints) and LHS can be factorized
18

---

Published as a conference paper at ICLR 2022
Σ [ u) - as (h -1 '(z)), Kh (Zz- u)) = [(Z,) - q(h (zz)).
(25)
where ạ is the marginal log-density of the components 2, when u is integrated out and does not need to be factorial.
Step 2
Now we do the following simplification of notations. Let h (2:) = [x)]( Denote
the first-order and second-order derivatives by superscript
9. (zvc (zc u.
(26)
Ozit
( (2t- }.
(27)
Ozy
and take derivatives of both sides of Eq. 25 with respect to Eje, we have:
(2 Σ qi (h, ( (2--) 0h, (2t)
(28)
Ozjt
Ohg (z:)
= (z:) E (2+))
(29)
Ozje
Denote the first order derivative of h-1 as u/ (2.) 8h (w.) and of (2:) is the second-order
8h 0zyt
derivative with respect to different component 2j't for any j * j'. Taking another derivative with respect to Zir on both sides of Eq. 29, the first term on LHS vanishes and we have:
Σ th (A. (2))(h ( u)ot (2)) (z.) + qf (AT ( u)of (as) 25 (30)
where eiو denotes the derivatives of RHS of Eq. 29 which does not depend on u. Same as (Hy- varinen al., 2019), collect all these equations in vector form by defining a;(y) vector collecting all entries o/ (2,) Ju{ (z+) for E [1, ni] and E 11. We omit diagonal terms, and by symmetry, take only one half of the indices. Likewise, collect all the entries up (zz) for [1, and j' E [1,j 1] in the vector b(z)). All the entries of "ه are in c(z,). These n(n - 1)/2 equations can be written single system of equations:
aa(i)) ( (z))(h- (z.rr)), u) bz(zz)g (z:))(( (zz))(("'(zz_))) u) c(z.). (31)
Now, collect the and b into matrix M:
M(z,) (a1 (z:)..... an (zi), bi(z),, ... bn(z))
(32)
Eq. 31 takes the form of the following linear system:
M(z, w( u) = c(z:),
(33)
where w are the vectors defined in the sufficient variability assumption, and w is defined for any input zІ. Notice that the RHS of the linear system does not depend the 2n + points u given for that 2, by the sufficient variability assumption.
fix z, and consider
Collect Eq. 33 above for 2n points starting from index 1:
M(z,) (w(ze; uz)..., ...,, w(2,, Uza)) (c(zz),... w(z, un))
(34)
19

---

Published as a conference paper at ICLR 2022
and collect the equation starting from index 0 for 2n points:
M(ze) (w uo)..... w(z,, 12n (c(zz)..... ..... (.)))...,,(z,uu)) uz))
(35)
Substract Eq. 35 from Eq. 34, we then have:
M(z:) [w u1) ((z,, uo).... ((z,, u2n) - w(Zr, uo)l = 0.
(36)
By the suffienct variability assumption, the matrix W that has linearly independent columns and square matrix is nonsingular. The only solution the linear system above thus:
M(z,) = (a1 (ze). an (2x). b,,z)., = O.
(37)
Following (Hyvarinen et al., 2019), (zz)) being zero implies no row of the Jacobian of h h- (z,) can have more than one non-zero entry. This holds for any z; By continuity the Jacobian and its in- vertibility, the non-zero entries in the Jacobian must be in the same places for all z;: If they switched places, there would have to be a point where the Jacobian singular, which would contradict the bijection properties of h derived in Section 1.3.1. This means that each h (z:) is function of only one Zkt for E n]. The bijection h also implies that each of the componentwise func- tions is invertible. Thus, have proven that latent variables identifiable up permutation and componentwise invertible transformations and temporally causal latent processes with conditions required by Theorem 1 are proved be identifiable from observed variables.
A.3.3 PROOF OF THEOREM 2
Theorem A.2 (Parametric Processes) Assume the vector autoregressive process in Eq. 38, where the state transition functions are linear and additive and mixing function is injective and differ- entiable almost everywhere. Lel B, E R=x= be the stale transition matrix lag т. The process noises ist are assumed to be stationary and both spatially and temporally independent:
1.
X4 x= 9Z+),
2t =
BBZI- + with. Cit ~
(38)
Nontinear mixing
T=1
Indepencient noise
Linear odditive transition
Here we assume:
1. (Generalized Laplacian Noise): Process noises Eit. ~ Pe; are mutually independent and follow agd: the generalized Laplacian distribution Pe; 2[[1/a.] exp (-Ailes]1*)) with os < 2;
2. (Nonsingular State Transitions): For at least one т, the state transition matrix B, is full rank.
Then the componentwise identifiability property of temporally causal latent processes is ensured.
Proof: The following proof is inspired by Theorem in (Klindt et al., 2020). The key differences are (i) allowing temporal causal relations B, among the sources instead of independent sources assumption, (ii) extending the single time lag restriction to multiple time lags case.
Identifiability on Related Sources Let us start from the simple case where the time lag 1. In this case, the transition dynamic in Eq. 38 can be simplified as
X = (Zz),
Z,
Bz, + 6z-
(39)
Using Eq. 15 and by applying the distributional forms of generalized Laplacian noise, have:
p(z:)
p(zz/z,-1) = p (z, (z,
p(h 1(1-1(zz) (z,))
(40)
Ml - 12 -N (zx) - BBh (z log p(z))
p(h- ((,))
20

---

Published as a conference paper at ICLR 2022
where M and N the constants appearing in the exponentials in p(zo|ze-1) and p(h (zz)( (z(-1)).
Taking the derivative w.n.t - on both sides, we obtain
0 0
B2
0 (1)) Bh (2c-1)
(41)
dz--1
dz--1
For the left hand of the Eq. 41, we can derive
0| 0z
2 Bz, l|
az
B2 11la
01 ) Bh- (2+-1)l18
0zz z2 Bz
Úz--1
òz--1
a||z:
BZi-1lla
Bh -1-11p
33h-122-11 (2t-1)
al
1t Bz
dh:-1
87--1
0llalla Ōz.
BZ
0||h (z,)
Bh
(z(-(( la
al|z; -Bz--1laa
(42)
De:
02:-1
821-1
11011 0
Øllcalla
Bhh -1
BZ 1la -allze
B
dec
87
0| (zz)
Bhh -1 (2t-1) 1)
(z Bz 2 22
Bzc-1 2-
ò2
Making the same derivation process on the right hand, we obtain
ad (z - 12 @ Zz B 1 1a-2B
=- =-a(h (2:) Bh. (2x- ④ I (z)) - Bh 13 a-2B Oh 1----(21
(43)
2:-1
For any z, choose z, Bz;-1 and thus the Eq. 43 can be written as:
(Z,) Bh (2,)() ④ 1h (2,) ) Bh (2.-))= a-22 Oh [2) 0.
(44)
2
Considering the nonsingularity of matrix B (by assumption) and bijection of h, we can derive
(h (zz) Bh 1 (24-1))
3 1h (2,) Bhh 1Bh-1(zz) (2 0
(45)
(h 1 (z:) -Bh, (z, 1) 14 (z.) Bh, - (2 0.
for all 1,.... d. Apparently h h- h-1(2)) Bh (z+-1) is the only solution, thus
h- h (Bz,-1) Bh (z)
(46)
Substitute Eq. 46 to the right hand of Eq. 40, we have
2 B 1 11 = |h (2t) - Bh 121-1)118
(47)
2t B2 = |1 (zx) - 11 (B=,
This indicates that h-1 preserves the a-distances between points. Since is bijective, then by Mazur-Ulam theorem Mazur & Ulam (1932), h must be affine transform. According to Theorem 2 in (Zhang & Hyvärinen, 2011), the model identifiable, which proves the theorem.
Extension to Multiple Time Lags We extend the result in Eq. 41 and have
Z Σ B,z, a Oz,-i
B,h (zz-7)lla
(48)
Oz.-;
where Zi-i is any lag latents with the limitation that the B, corresponding z,- is of full rank. Following the same above-ment noove-mentioned derivation process, we can obtain
(ze B.hh-n - Σ B, a-
(49)
L =(h (2x) - Σ B,h - 0) -1())= - Σ
I. B, -1 (Zz-7))a
Oh ((c-i)
T=1
T=1
21

---

Published as a conference paper at ICLR 2022
For any 2, choose E/== B, thus the Eq. 49 can be written as:
(h (z)) Σ B (zc-1) 1) 1] 1/ (z=) Σ B (z:-,)) =B, Oh (zi-1)
0. (50)
Ze-
As mentioned above, h (z=) = =Er=l B, B, is the only solution, thus
h- Σ B, SB B (Z(-r).
(51)
Following the same procedure in the simple case, the theorem is proven.
A.4 COMPARISONS WITH EXISTING THEORIES
The closest work to ours includes (1) PCL (Hyvarinen & Morioka, 2017), which exploited tempo- ral constraints to separate independent sources, (2) SlowVAE (Klindt et al., 2020), which leveraged sparse transition of adjacent video frames separate independent sources, and (3) iVAE (Khe- makhem et al., 2020), which leveraged the nonstationarity by the modulation of side information on the prior distribution p(z) u) conditional factorial latent variables. Our work extends the theories to the discovery of the conditional independent sources with time-delayed causal relations in between by leveraging nonstationarity, or functional and distribution forms of temporal statistics. To the best of our knowledge, this is one of the first works that successfully recover time-delayed latent processes from their nonlinear mixtures without using sparsity or minimality assumptions.
PCL The sources 2st in PCL were assumed be mutually independent (see Assumption of Theorem I in PCL). In contrast, we allow the sources to have time-delayed causal relations in between, which much more realistic in real-world applications. They further assumed the sources are stationary, while we allow nonstationarity in the nonparametrie setting (the nonstationary noise assumption). The underlying processes of PCL are described by Eq. 52:
log (2g.((,,,).)))) G(a,, p
log ((2,,11,,,)11) (2, r(E,,-1))) + const.
(52)
where is some non-quadratic function corresponding to the log-pdf of innovations, < 1 is re- gression coefficient, some nonlinear, strictly monotonic regression, and À is positive precision parameter. Both theorems of our work extend the theory to the discovery of the conditional inde- pendent sources with time-delayed causal relations in between. Furthermore, in the nonparametric process condition, we do not restrict the functional and distributional forms of underlying transitions. Our proposed nonparametric condition naturally includes PCL as a special case.
VAe Inspired by slow feature analysis, Slow assumes the underlying sources to have identity transitions with generalized Laplacian innovations described in Eq. 53:
p(z./1-1 II i=1 2T(1/a) exp 1app-Ales,t =,, with a <
ad
(53)
Our proposed parametric condition (Theorem 2) in Eq. is natural extension to the Laplacian innovation model above by allowing time-delayed vector autoregressive transitions in the latent process with multiple time lags. Consequently, temporally causally-related latent processes with linear transition dynamics can thus be modeled and recovered from their nonlinear mixtures with our parametric condition.
Similar to TCL (Hyvarinen & Morioka, 2016) and GIN (Sorrenson et al., 2020), iVAE ex- ploits the nonstationarity brought by the side information (i.e., class label) on the prior distribution of latent variables As one can see from Eq. 54, the latent variables are conditionally indepen- dent, without causal relations in between while both of our theorems consider (time-delayed) causal relations between latent variables. In addition, iVAE exploits the nonstationarity brought by side information (i.e., class label) on the prior distribution latent variables z. On the contrary, our non- parametric condition, instead of relying the change in the prior distribution of latent variables,
IVAE
22

---

Published as a conference paper at ICLR 2022
exploits the nonstationarity in the noise distribution, which is more natural in real-world datasets. Finally, iVAE assumes modulated exponential families in Eq. 54 while our nonparametric condition (Theorem 1) allows any kinds of modulation by side information without those strong assumptions on the transition functions or distributions.
(1(z)
PT. (z)u = II (u)
(. exp
(54)
j=1
In terms of architecture innovations, remove distributional and functional form constraints of iVAE, we design a novel causal transition prior network for nonparametrie transitions by injecting the IN condition inside reparameterization trick, resulting in an efficient scoring mechanism of tran- sition prior which only needs to compute the determinant of low-triangular Jacobian matrix. This module was never seen in previous work.
A.5 CONNECTING THEORIES TO MODEL
As one can see from the proofs in Appendix A.3, what have been assumed for the estimation frame- work are the conditional factorial properties of of temporal nonstationarities through nonstationary noises. The conditional factorial properties have been injected using the reparameterization trick (Eq. 7) with the IN condition in causal transition prior and the enforcing of spatiotemporal independence of estimated residuals through contrastive learning. The nonstationary noises are modeled with flow-based density estimators. We share the weights of the other modules (e.g., encoder, transition function, decoder, inference network, etc.) across nonstationary regimes while using separate flow models estimate the density of residuals and evaluate the prior scores in each regime. We also use componentwise flow models so the learned residuals will not interact with each other the estimation framework. Finally, nonparametric processes, we warm-start the flow models to generate standard Gaussian noise. In parametric pro- cesses, the flow models are initialized to generate standard Laplacian noise. Note that the other assumed conditions in the two theorems, such sufficient variability and nonsingular state transi- tions, are data properties and do not need to be encoded as constraints in the estimation framework.
u) where 2, h 1--1 (zz) and the model
B EXPERIMENT SETTINGS
B. SYNTHETIC DATASET
Seven synthetic datasets, including two datasets (NP and VAR) which satisfy our assumptions, and five datasets, which violate each of the assumption in the proposed theorems, are used in this paper. We set the latent size 8 and the lag number of the process L 2. The mixing function g is random three-layer MLP with LeakyRel.U units.
Nonparametric (NP) Dataset For nonparametric processes, we generate 1 150,000 data points ac- cording Eq. 2. In particular, noises 6e are sampled from i.i.d. Gaussian distribution with variance modulated by 20 different nonstationary regimes. In each regime, the variance entries are uniformly sampled between 0 and 1. A 2-layer MLP with LeakyReLU units is used as the state transition function fi. When we need sparse causal structure for visualization, random binary mask is added to the input nodes.
Gaussian additive noise model as the latent processes. The
(Violation) Insufficient Variability For this dataset, we create datasets that violate the nonsta- tionary noise condition and sufficient variability by restricting the number of nonstationary regimes observed in the NP dataset. When only one regime is observed, we violate the nonstationary noise condition by using stationary noise. Furthermore, we vary the number of the observed regimes |u| E {1,5, 5, 10, 15, 20} to assess the impacts of variability on the recovery of nonparametric pro- cesses.
Parametric (VAR) Dataset For parametric processes, generate 50,000 data points according to Eq. The noises e are sampled from i.i.d Laplacian distribution (a = 0.1). The entries of state transition matrices B, are uniformly distributed between [-0.5, 0.5).
23

---

Published as a conference paper at ICLR 2022
(Violation) Low-rank State Transition For this dataset, the transition matrix B, in Eq. 4 is low- rank instead of full-rank. The datasets are created following the steps in the VAR dataset, but we restrict the rank of state transition matrix B, to 4 and time lag 1. The full matrix rank is 8.
(Violation) Gaussian Noise Distribution For this dataset, the noise terms in Eq. 4 follow the Gaussian distribution (o, 2) instead of Generalized Laplacian distribution (a, 2). In particular, the noise terms are sampled from i..d. Gaussian distribution (o = 0.1).
(Violation) Regime-Variant Variant Causal Relations For regime- variant causal relations, we generate 240,000 data points according to Eq. 55:
L Σ
g(Ze),
B with Pry
(55)
The noises Ex are sampled from i.idd Laplace distribution (a = 0.1). In each regime u, the entries of state transition matrices B uniformly distributed between [-0.5, 0.5).
(Violation) Instantaneous Causal Relations For instantaneous causal relations, we generate 45,000 data points according to Eq. 56:
I B + with Ca ~ Pes
x; = g(z)), Zt Az;
(56)
T=1
where matrix A is random Directed Acyelic Graph (DAG) which contains the coefficients of the linear instantaneous relations. The noises Gir are sampled from i.i.d. Laplacian distribution with
0.1. The entries of state transition matrices B, are uniformly distributed between [-0.5, 0.5).
B.2 REAL-WORLD DATASET
Three public datasets, including KiTTiMask, Mass-Spring System, and CMU MoCap database, used. The observations together with the true temporally causal latent processes are showcased in For CMU MoCap, the true latent causal variables and time-delayed relations are unknown.
Fig. B.1.
(+)
(b)
(c)
Figure B.1: Real-world datasets: (a) KiTTiMask video dataset of binary pedestrian masks, (b) Mass-Spring system is video dataset with ball movement rendered in color and invisible springs, and (c) CMU MoCap is 3D point cloud dataset of skeleton-basee signals.
The KiTTiMask dataset consists of pedestrian segmentation masks sampled from the autonomous driving vision benchmark KiTTi-MOTS. For each given frame, the position (vertical and horizontal) and the scale of the pedestrian masks are set using measured values. The difference in the sample time (e.g., At 0.15s) generates the sparse Laplacian innovations between frames.
KiTTiMask
Mass-Spring System The Mass Mass-Spring system is a classical physical system that several objects are connected by some visible/invisible spring, which follows Hooke's s law. In this work, we consid- ered the system with five degrees of freedom and made linearization on the state without calculating the Euclidian distance between objects. Thus, there are ten causal relations, six which were set connected, and the other four were disconnected. The rest length of the spring was uniformly distributed between [1, 10), and the stiffness of the spring relation was set as 20. The action was 1.
a, 300e,, where e followed the Laplacian distribution with mean μ 0 and variance We assumed there was no damping the system and randomly assigned the objects in different positions at the beginning of each episode.
24

---

Published as a conference paper at ICLR 2022
cmu edu/) / is open-source human tion capture dataset with various motion capture recordings (e.g., walk, jump, basketball, etc.) per- formed by over 140 subjects. In this work, we fit our model on 12 trials of "walk" recordings (Sub- ject 7). Skeleton-based measurements have 62 observed variables corresponding to the locations of joints (e.g., head, foot, shoulder, wrist, throat, etc.) of the human body at each time step.
CMU MoCap CMU MoCap (http://mocet
B.3 EVALUATION METRICS
SHD: Structural Hamming Distance We use SHD (de Jongh & Druzdzel, 2009) to measure the distance between two causal graphs. It computes the number of edge insertions, deletions, flips in order to transform one graph to another graph. SHD one variant of Minimum Edit Distance (MED) in causal discovery area by allowing only insertions, deletions, and flips of edges.
MCC: Mean Correlation Coefficient MCC is standard metric for evaluating the recovery of latent factors ICA literature. MCC first calculates the absolute values of correlation coefficient between every ground-truth factor against every estimated latent variable. Depending whether componentwise invertible nonlinearities exist in the recovered factors, Pearson correlation coeffi- cients or Spearman's rank correlation coefficients can be used. The possible permutation is adjusted by solving a linear sum assignment problem in polynomial time on the computed correlation matrix.
In this work, we the Pearson correlation coefficient for the VAR processes and Spearman's correlation coefficient for the NP processes.
C IMPLEMENTATION DETAILS
In this section, first provide the network architecture details of LEAP, The hyperparameter selec- tion criteria and sensitivity analysis results are presented. The training settings are summarized.
C.I NETWORK ARCHITECTURE
We summarize our network architecture below and describe it in detail in Table C.I and Table C.2.
(1,2) MLP-Encoder and MLP-Decoder: These modules are used for the synthetic and mo- tion capture datasets. They are composed of a series of fully-connected neural networks with LeakyReLU the activation function. The universal approximation theorem guarantees that model can approximate the mixing function. The encoder maps the raw observations into features, while the decoder maps the latent variables back the inputs.
(3,4) CNN-Encoder and CNN-Decoder: For the KiTTiMask dataset, vanilla CNNs are used for both the encoder and decoder. For the Mass-Spring system dataset, the time-delayed causal vari- ables use objects as the building blocks to factorize the scene. The object-centric representations contain object locations and some other attributes (e.g., color, size, etc.). We thus two separate CNNs, one for extracting visual features (see Feature Extractor in Table C.2) and the other for locating object locations with spatial softmax unit (see Keypoint Predictor in Table C.2). The decoder retrieves object features from feature maps using object locations and reconstructs the scene (see Refiner Table C.2).
We apply the bidirectional Gated Recurrent Unit (GRU) (Cho et al., 2014) to preserve both the past and the future information. It processes the input sequence x, in both directions: one for the forward pass and one for the backward pass. We denote the forward and backward embeddings H and H. The other inference module (see DemporalDynane in Table C.1) uses the sampled/inferred past latent variables to compute the posterior of z, We insert skip-connections (He et al., 2016 60 between the two inference modules avoid the vanishing gradient problem and obtain better model convergence performances. Note that for the first iemporally-earliest latent variables in the sequence, there is time-delayed information, and use isotropic Gaussian (O, 1) as their prior distributions. The prior of the remaining sequence is evaluated with the learned transition prior network described below.
(5) Inference Network:
(6,7) Causal Process Prior Network This module contains three components. (i) Inverse tran- MI.s to compute the estimated noises. For sition functions. For the NP transitions,
25

---

Published as a conference paper at ICLR 2022
the VAR transitions, group of state transition matrices, one for each time lag, is used. (ii) log (Idet (J))) computation. For the NP transitions, the Jacobian matrix entries dri Dā. are com- puted using torch. autograd. Functional jacobiar method. The log determinant is then evaluated by summing over log transformations of absolute values of Jacobian terms for 2n-- For the VAR transitions, because of the additive noise assumption, the log determinant is directly (iii) Spline flow model. Componentwise spline flow models use monotonic linear rational splines transform standard Gaussian distribution the estimated noise distribution. We use eight bins for the linear splines and set the bound as five so data points lying outside [-5,5] are evaluated using. (001 1) directly while the data points within the region evaluated by spline flow models. For the nonparametric processes, we always warm-start the spline flows by training it on dataset of standard Gaussian noises with steps=5000 and learning rate=0.001. For the para- metric processes, we warm-start it instead on dataset of standard Laplacian noises. All the three components of the transition prior network are be learnable during the VAE updates.
Table C.I:
Architecture details. BS: batch size, length of time series, dim: input dimension, 2_dim: latent dimension, LeakyReI.U: Leaky Rectified Linear Unit.
Configuration
Description
Output
MLP-Encoder
Encoder for Synthetic/MoCap Data
Input: X1:T
Observed time series
BS x T x i_im
Dense
128 neurons, LeakyReLU
BS T x 128
Dense
128 neurons, Leaky ReLU
x T x
Dense
128 neurons, Leaky ReLU
BS x T x 128
Dense
Temporal embeddings
BS T x z 2_dim
MLP-Decoder
Decoder for Symthetic/MoCap Data
Input: 81:T
Sampled latent variables
BS T x z_dim
Dense
128 neurons, LeakyReLU
T x 128
Dense
128 neurons, LeakyReLU i d_d neurons, reconstructed &
BS x T x 128
Dense
BS x T x i_dim
Inference Network
Bidirectional Inference Network
Input GRUInference
Sequential embeddings
BS x T x z z_dim
Bidirectional inference
T x 2+z_dim
TemporalDynamics
Use past (2,.)) to infer postcriors zı Skip-connection of the two inferences
BS x T x 2+z_dim
ResidualBlock
BS x T x 2+z dim
Bottleneck
Compute mean and variance of posterior
Reparameterization
Sequential sampling
21T
6. Causal Process Prior (VAR)
Linear Transition Prior Network
Input
Sampled latent variable sequence 21TT Compute estimated residuals é. Score the likelihood of residuals
x z_dim
InverseTransitiie
BS x T x z_dim BS
SplineFlow
Causal Process Prior (NP)
Nonlincar Transition Prior Network
Input Transstorann
Sampled latent variable sequence 2.. Compute estimated residuals éis
BS x T x z_dim
BS T x z
JacobianCompute
Compute log (Idet (J)))
BS BS
SplineFlow
Score the likelihood of residuals
Model Ablations We start with BetaVAE and add our proposed modules successively as model variants. When the causal process prior network is added the baseline, this model variant equipped with the inference network and learned inverse transition functions. However, during the estimation of KL divergence in the causal process prior network, we use Mean Squared Error (MSE) directly, which corresponds stationary Gaussian noise distribution, to replace the flow density esti- mators that evaluate the prior likelihood scores across nonstationary regimes. This variant shows the contributions of the learned causal transition functions. Furthermore, when the nonstationary flow estimator added to the variant, this variant is implemented by removing the noise discriminator from L.AP while not changing any training settings.
26

---

Published as a conference paper at ICLR 2022
Table C.2: Architecture details on CNN encoder and decoder. BS: batch size, T: length of time series, h dim: hidden dimension, z d_di:: latent dimension, F: number of filters, (Leaky)ReLU: (Leaky) Rectified Linear Unit.
Configuration
Description
Output
3.1.1 CNN-Encoder
Feature Extractor
Input: x1.7
RGB video frames
BS X T T X
X
3
64
64
Conv2D
F: 16, BatchNormm2D, LeakyReLU
16 x
x
Conv2D
F: 16, BatchNorm22, JeakyReI.U
BS x
T x
16 x 64 x 64
Conv2D
F: 32, BatchNormn2D, LeakyReLU
BS T x 32 x 32 x 32
Conv2D
F: 32, BatchNorm22, LeakyRel.J
x
x
x 16
Conv2D
F: 64, BatchNorm22, LeakyRel.U
BS X T x 64
x
x 16
Conv2D
F: 5 number of objects
BS
T x
5
16
16
3.1.2 CNN-Encoder
Keypoint Predictor
Inpul: x1:7
RGB video frames
BS T x 3
64 64
Conv2D
F: 16, BatchNormm2D, LeakyReLU
BS T X
16
x
x
Conv2D
F: 16, BatchNNrrm2D, LeakyReLU
BS X T X
16
x 64 x 64
Conv2D
F: 32, BatchNorm22, LeakyReI.U
BS X T X
32
x 32 x 32
Conv2D
F: 32, BatchNorm22,, LeakyReLU
T X
x
x
CDnv2D
F: 64, BatchNorm22, LeakyRel.U
BS X
T X
64
X
16
X 16
Conv2D
F: 5 number of objects SpatialSoftmax, lim-[-1.1.-111]
BS T X 5
16
16
Conv2D
T X
2
3.2 KiTTiMask-EEecaa
Mask Encoder
Inpul: X1.7
Semantic-segmented video frames
BS T x
64
64
Conv2D
F: 32. BatchNNorm2D, ReLU
BS x T x 32
x 32 x 32
CDnv2D
F: 32, BatchNorn22, ReLU
BS T X 32 X 16 X 16
Conv2D
F: 64, BatchNorm22, ReLU
BS T x 64 X
8
Conv2D
F: 64, BatchNorrm2D, ReLU
BS x T x 64 x 4 X
Conv2D
F: h_dim, BatchNorm22, ReLU
BS T X h_dim
X x
Dense
2+z dim neurons, features R1-
T x 24z dim
4.1 CNN-Decoder
Refiner
Input: Z1:r'
Sampled latent variable sequence F: 64, Batchhorm22, LeakyRel. ReLU F: 64, BatchhNorm2D, LeakyReLU
BS T x 64 x 2 x
Conv Transpose2D
BS T x
x
Conv2D Conv Transpose2D
BS x T x
64 x 32 x 32
F: 32, BatchNorm22, LeakyReLU
BS T
X
x 64
Conv2D
F: 32, BatchNorm22D, LeakyReLU
BS x
T x
32
X 64
x 64
Conv2D
F: 3, estimated scene &
BS x
T
X 3
64
64
4.2 KiTTiMask-Deeodee
Mask Decoler
Input: 21.7 Dense
Sampled latent variable sequence
BS x T x Z_dim
h dim neurons
BS T X h dim
Conv Transpose2D
F: 64, BatchNorm22, ReLU
T X
X 4
Conv Transposc2D
F: 64, BatchNorm22,, ReLU
BS X T X 64 X 8 x 8
Conv C Transpose2D
F: 32, BatchNormm2, ReLU
BS T x 32 x 16 x 16
Conv Transpose2D
F: 32, BatchNorm22, ReLU
T x
x
x
Conv TannTrassposDD
F: 1, estimated scene &
BS T X 1
64 64
C.2 HYPERPARAMETERS AND TRAINING DETAILS
We describe the hyperparameter selection criteria and discuss the impacts of hyperparameter values on the model performances. The training details are provided.
C.2.1 HYPERPARAMITES SELECTION AND SENSITIVITY ANALYSTS
The hyperparameters of LEAP include [8, o), which the weights of each term in the augmented ELBO objective, as well as the latent size n and maximum time lag L. We use the ELBO loss on the validation dataset select the best pair of 7, because low ELBO loss always leads high
27

---

separate Published as a conference paper at ICLR 2022
14
025
9m-3
223
660
0.5
0.3
300
200
303
(a) MCC trajectory with different B and 7
(b) Visualization the final MCC.
1.0
3.0
0.5
lipe3
0.4
Se
0.2
0.0
60
(c) MCC trajectory with different a.
(d) MCC with different latent size and time lag.
Figure C.I: Impacts of hyperparameters on VAR dataset.
MCC. We always set larger latent size than the true latent size. This critical in video datasets because the image pixels contain more information than the annotated latent causal variables, and restricting the latent size will hurt the reconstruction performances. For the maximum time lag L, we set it by the rule of thumb. For instance, we use L = 2 for temporal datasets with latent physics process.
However, it is known (Mita et al., 2021) that the performances of VAEs might change drastically as a function of the regularization strength. We thus conduct sensitivity analysis on the impacts of identifiability performances. We report the MCC scores for the synthetic NP and VAR datasets on a hyperparameter grid. We found that the values 3 and y have a larger impact the identifiability results, while the effects of a are relatively smaller. Furthermore, have verified the robustness of our approach under different maximum time lags and latent size settings. The final MCC scores only show marginal differences, indicating that our approach is robust to the choices of n and L. In summary, the performance of our approach robust to the values of some of the hyperparameters, and for the remaining hyperparameters, validation data to set their values.
hyperparameters
Parametric (VAR) Dataset We have performed a grid search of 3 E [3E-4, 3E-3, 3E-2] and y 19E-4, 9E-3, 9E-2] and reported the results in Fig. C.)( The best configuration [8,7] [3E-3, 9E-3]. We plot the final MCC score as a function of the value of the two hyperparameters [5E-7, 1E-6, 1E-5) with the optimal (8,7) value. The optimal configuration for a is 1E-6 as shown in Fig. C.1(c). Furthermore, we verify the robustness of our approach under different time lags
Fig. Cb. For a, we compare the MCC scores under different
[2,3, 4) and latent dimen- sions E [4, 6, 8) with [ ol = [3E-3, 9E-3, 1E-6) and the results are shown in Fig. C.l We can see that the final MCC scores all cases are around 0.9 with marginal differences, indicating that latent recovery performances of our approach is robust the choices of n and L.
Nonparametric (NP) Dataset Similarly, we have performed grid search of 3 [2E-4, 2E-2] and 7 E 22-3, 2E-2, 2E-1] on the NP dataset and reported the results in Fig. C.2(a). The best configuration is [8,r] /2-3,, 2E-2]. The final MCC scores under parameter grids of (] are shown in Fig. C.2(b). The optimal configuration for o is 1E-6 in the search space [1F-7, 1E-6, 1E-5) with the optimal [3,r] value, shown in Fig. C.2(c). We verify the robust-
28

---

Published as a conference paper at ICLR 2022
ness in terms of latent size n [6. 7, 8] and the maximum time lags L [1,2,3] with the optimal configuration 13,%% Øj = [2E-3, 1E-6] and the results are shown in Fig. C.2(d). The final MCC scores in all cases are around or higher than 0.85 with marginal differences.
1.2
6.00
2o 2012
355
1/8 8ig 12
1/8%
300
200
303
(a) MCC trajectory with different B and 7-
(b) Visualization the final MCC.
1.0
3.0
0.5
6.4
lge2
0.4
1c
p= 16-6
02
16
6.0
0.0
(c) MCC trajectory with different a.
(d) MCC with different latent size and time lag.
Figure C.2: Impacts of hyperparameters on NP dataset.
C.2.2 TRAINING
Training Details The models were implemented in PyTorch 1.8.1. The VAE network is trained using AdamW optimizer for maximum of 200 epochs and early stops if the validation ELBO loss does not decrease for five epochs. A learning rate of 0.002 and a mini-batch size of 32 are used. For the noise discriminator, we use SGD optimizer with a learning rate of 0,001. We have used four random seeds in each experiment and reported the mean performance with standard deviation averaged across random seeds.
Computing Hardware We used machine with the following CPU specifications: Intel(R) Core(TM) 17-7700K CPU @ 4.20GHz; 8 CPUs, four physical cores per CPU, total of 32 logi- cal CPU units. The machine has two GeForce GTX 1080 Ti GPUs with IIGB GPU memory.
Training Stability We have used several standard tricks to improve training stability: (1) we use slightly larger latent size than the true latent size for real-world datasets in order to make sure the meaningful latent variables are among the recovered latents; (2) we use Adam optimizer as regularizer to prevent training from being interrupted by overflow or underflow of variance terms of VAE; (3) we use a larger learning rate for the VAE than for the noise discriminator to prevent extreme extrapolation behavior of discriminator.
D ADDITIONAL EXPERIMENT RESULTS
D.1 COMPARISONS BETWEEN LEAP AND BASELINES ON CMU-MOCAP DATASET
Because the true latent variables for CMU-Mocap unknown, visualize the latent traversals and the recovered skeletons, qualitatively comparing our nonparametric method with baselines in terms of the how intuitively sensible the recovered processes and skeletons are.
29

---

Published as a conference paper at ICLR 2022
Latent variable index
P
Latent variable value
(a) BetaVAE
(b) SlowVAE
(c) LEAP
Figure D.I: Latent traversal comparisons between LEAP and the baselines. LEAP represents the data with aulally-related factors, thus can represent the data with only much fewer latent variables (three vs eight) with smooth transitions dynamics. Video demonstrations are in: https: /bit. 1y/ SkEkh..
Latent Traversal We fit LEAP and the baseline models using the same latent size n = 8 and the maximum time lags 2. As shown in Fig. D.1, LEAP represents the data with causally-related factors, thus explaining the data with fewer latent variables and smooth transitions dynamics. Only three latent variables are in fact used by LEAP and while the other five latent variables only encode random noise seen from the video demonstration. BetaVAE and Slow VAE, however, need to use all the latent variables to represent the data. Furthermore, we find the three latent variables discovered by LEAP encode pitch, yaw, and roll rotations of walking cycles, which is close to how human beings perceive walking movement.
Recovered Skeleton As shown Fig. D.2, LEAP recovers the cross relations between causal variables while BetaVAB and VAE only recover independent relations. The latent traver- sals of LEAP have shown that the three recovered latent variables may be the pitch, yaw, and roll rotations of the walk cycles. Therefore, the results of approach indicate that the pitch (e.g., limb movement) and roll (c.g., shoulder movement) are causally-related while yaw has independent dynamics, which closer to reality than the independent transitions discovered by VAE and Slow VAE.
D.2 MASS-SPRING SYSTEMS
We render the recovered latent variables using keypoint heatmaps in Fig. D.3(a). The learned rep- resentation successfully disentangles five objects the scene, and the latent variables represent the horizontal and vertical locations of the balls. We further visualize the recovered skeletons from the estimated state transition matrices in Fig. D.3(b). The recovered skeleton is consistent with the underlying processes described in Fig. Bbb(b) with SHD=00.
30

---

Published as a conference paper at ICLR 2022
BetaVAE Estimated Skeleton
SlowVAE
LEAP
Latent variables
Estimated Skeleton
Estimated Skeleton
Time-delayed latent variables
Figure D.2: Comparisons between LEAP and the baselines in terms of skeleton recovery. LEAP recovers cross relations between causal variables while baselines can only recover independent re- lations.
3
Heatmmp for
Heettmm bal
Heetmmp bell
(a) Latent variahles for fixed video frame.
(b) Recovered causal skeletons.
Figure D.3: Visualization recovered latent variables and the estimated skeletons for Mass-Spring
system dataset.
E EXTENDED RELATED WORK
Temporal dependencies and nonstationarities were recently used as side information u to achieve identifiability of nonlinear ICA on latent space z. Hyvarinen & Morioka (2016) proposed time- contrastive learning (TCL) based on the independent sources assumption. It gave the very first identifiability results for nonlinear mixing model with nonstationary data segmentation. Hyvari- nen Morioka (2017) developed a permutation-base separate independent sources using temporal dependencies. Their approach learns to discriminate between true time series and permuted time series, and the model is identifiable under the uniformly dependent assumption. Hälvå & Hyvarinen (2020) combined nonlinear ICA with Hidden Markov Model (HMM) to automatically model nonstationarity without the need for manual data segmen- tation. Khemakhem et al. (2020) introduced VAEs approximate the true joint distribution over observed and auxiliary nonstationary regimes. The conditional distribution in their work p(z|u (u) is assumed to be within exponential families achieve identifiability on the latent space. more recent study in causally-related nonlinear ICA was given by (Yang et al., 2021), which introduced linear causal layer to transform independent exogenous factors into endogenous causal variables.
contrastive (PCL) learning framework to
31


Reviews:
[{"review_id": "YGgFlfQ9OSw", "reviewer": "Reviewer_Qhdx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have some concerns:\n1. The proof of the theorems are not clear. E.g., what is the condition u in Eq. (2) and the following many equations? \n2. about the network structure: input is x_0, x_1, ...., x_T, and the latent is z_0, z_1, z_T, and the reconstruct is x_hat_0, ..., x_hat_T. The \"time-delayed\" is not clear in this structure. If there is time delay, how the x_0 can be reconstructed, and how is the reconstruction loss is calculated?\n3. In Fig. 2, it is not clear what's the importance of each part of the model. How does the \"C\" part connect to the proposed two conditions? As the model seems big, what about the training stability?\n4. The hyperparameters of the loss function are not clear. How to select the hyperparameters and their importance. The importance of each term of the loss is not very clear. \n5. The relation of using masked input and the \"sparse\" is not clear. The mask is for sparse latent process, while it is also stated in the paper the method does not rely on sparsity in latent processes.\n6. The structure of using \"contrastive learning\" and \"flow\" to ensure the identifiability are already exist in previous work by Khemakhem et al. I am not sure of the comparison with the previous work and the exact novelty. ", "summary_of_the_paper": "This paper propose two new conditions for nonlinear identifiability of temporal processes. The idea is very promising: discovery of conditional independent sources with time-delayed causal relations in between. \nThe proof and the code seem good.", "summary_of_the_review": "The idea is very promising but some technical details are not very clear. Besides the proofs in the appendix, it is recommended to describe more about the architecture and implementation details in the appendix too, if length limited. It'e better also to add some discussion about how the conditions are satisfied in real applications and what to do if the conditions are not satisfied", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KgqmzMrEvRB", "reviewer": "Reviewer_7eEa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- In general, the topic of deriving rigorous conditions for the identifiability of temporal causal LVMs, and developing novel methods based on these conditions, is very interesting and relevant to the ML community. \n-To my knowledge the setup considered in the article is novel (though see the cons below).\n-The theoretical derivations seem correct (although I only glanced through the proofs in the supplement).\n-The experiments are sufficiently thorough, including synthetic experiments and 3 real-world data sets.\n\nCons and questions to clarify:\n- It seems that the (independence) assumptions, proofs, and the methodological ideas (e.g. contrastive learning to ensure independece) are rather close to previous works (e.g. by Klindt or by Hyvärinen and Morioka). The previous works have been properly cited in the text. Nonetheless, it would be beneficial for me to see a more concrete side-by-side comparison, to see where the novelty is located exactly (e.g. in the Supplement). For example, Eq. (2) and (4) very clearly present the current setup. Would it be possible to compare these with similar concise summaries of the previous setups?\n-The method comes with many hyperparameters and it is not clear what their values are (e.g. beta, gamma, sigma in Eq. 11) or how they were selected (e.g. the latent dimension and the time lag in the experiments). Hence, judging how well the method really behaves in practice is difficult. \n-Some assumptions are a bit abstract, for example the \"Sufficient variability\". How would a user of the method know if his/her data satisfy these assumptions? How do you know these assumptions are satisfied in your real world examples, and what happens if some of the assumptions are violated?\n-The clarity of the presentation could be improved. For example, some figures are so small that it is impossible to make anything out of them when printed out (e.g. Fig. 8). Another concrete example: in the preliminaries of the proof for identifiability (section A.1) a sentence: \"Since the learned unmixing function...\" Why is g hat called the \"unmixing function\" here (and not elsewhere), what is g* and what is h (the sentence breaks in the middle before explaining h), and why the \"we can assume...\"? Similar small clarity and grammatical issues make the reading a bit cumbersome at places, although the technically the papers appears correct in general.\n\nVery minor:\n- Can you clarify where Eq. (6) comes from?\n-In Fig 9, on the x-axis you have z_{t-1}^2 and z_{t-2}^2 twice. Is this a typo?\n", "summary_of_the_paper": "The paper considers the learning of a temporal latent causal process: x_t=g(z_t), where x_t are observations, z_t are the underlying latent variables, and g is unknown. The paper makes two contributions: 1) presenting conditions under which the process can be identified (up to a permutation of the latent variables and their componentwise invertible transformations), and 2) developing a training framework that enforces the assumed conditions. The first contribution is theoretical and the second methodological. In addition, the paper presents empirical evidence for the ability of the method to recover the underlying causal process using synthetic experiments and real-world data.\n\nTwo cases are considered: 1) non-parametric, where a non-parametric transition is assumed in the latent space, and 2) parametric, where linear additive transitions are assumed. The main assumption for identifiability is the independent noise (IN) condition, which states that noise for the dimensions of z_t are independent of each other and across time steps t. This assumption is enforced through a contrastive approach during training. In addition, in the non-parametric case the noise is assumed non-stationary, such there exists sufficiently many auxiliary variables that affect the distribution of the non-stationary noise.\n", "summary_of_the_review": "An interesting paper with relevant results on the identifiability of latent causal processes. I hope the authors could clarify my above concerns in their rebuttal.\n\nPOST-REBUTTAL UPDATE:\nI found the authors had done a careful job addressing my original concerns. I don't have any remaining significant concerns. I find the article interesting and significant, and I support its acceptance at this stage. I will increase my score from 5 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J7pjobd4Xnb", "reviewer": "Reviewer_hJy9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is very promising and represents a step forward in the causal inference of temporal processes. However, there is a lack of clarity in the derivation of the model architecture that prevent future readers to use this paper at its full potential. I would be happy to increase my score if this issue is resolved.\n\n### Model details\n\n- The model derivation lacks clarity. I would not be able to reproduce the proposed architecture based only on the information given in the paper (appendix included). I think this is quite critical, even if the code is available online. In particular, Figure 2 has tildas on z, which are not to be found in the derivations of section 3 and this confusion seems still present in 3.1.1. I really encourage the authors to invest effort in making this architure more understandable, and especially highlight how each of the assumptions from theorems 1 and 2 are embodied. \n\n- Again regarding clarity, the $\\mathbf{u}$ are never defined nor properly introduced anywhere. This makes it harder to understand assumption 2 of Theorem 1. The IN condition, similarly, lacks a proper introduction (I assume it is the independent noise assumption).\n\n- The sufficient variability condition is still a bit cryptic to me in terms of its implications. I think it would be nice to expand a bit on that as well.\n\n### Experiments\n\n- For the recovery of the causal relations, you compare the causal skeletons of the true and inferred process but it could you give a measure of the discrepancy between the causal graphs ? Such as minimum edit distance (up to a permutation of the true causal graph) ?\n\n- The performance of the approach are very good compared to the baselines on the synthetic data. However, the data is generated as to fit the assumptions of Theorem 1 and 2. I would be good to also report the results of the baselines on the real-world datasets when the conditions are more general. Indeed, in this case, it's not clear if the proposed approach would as neatly outperform the baselines.\n\n- In \"Towards generalizing conditions\" and in the conclusion, you seem to state that the only limitation of the method is that is assumes constant causal graph with no instantaneous relations. This is without counting on the assumptions laid out in the theorems. In particular, non-stationary noise and sufficient variablity. Of course, you need some conditions to be able to recover something but the impact of violating those conditions would be worth investigating.\n", "summary_of_the_paper": "This papers proposes new conditions for identification of latent temporal causal processes, for both non parametric and parametric processes. They then propose an architecture emboyding the derived conditions such as to recover the latent temporal processes. The approach is shown to give better performance than compared baselines on synthetic data and to show good performance on non-synthetic datasets.", "summary_of_the_review": "Very promising paper but there is some lack of clarity in the descriptions of the architure used, preventing future readers to use this paper at its full potential. I would be happy to increase my score if this issue is resolved. The baselines should also be run on the real-world datasets for comparison with the proposed approach when the data generating process does not exactly matches the assumptions made in this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9mo3xWN7Rvg", "reviewer": "Reviewer_x73T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please list both the strengths and weaknesses of the paper. When discussing weaknesses, please provide concrete, actionable feedback on the paper.\n\nThe specific strength of this paper is described above. Although technical contributions are derived from mixing the previous techniques, the problem setting seems to be completely new (but I am not an expert in this field), and experiments were sufficiently performed. Partly I have some specific comments as follows. \n\n1. Sections 1 and 2: Parent (node) may be explicitly unclear. Is it the cause of the factor z? \n\n2. Sections 1: follow follow\n\n3. Definition 1: p_{\\epsilon} is not defined in the introduction.\n\n4. Fig 3c: It was unclear to me how to see Fig 3c. Is the negative correlation acceptable even if Figs 2a and 2b do not seem to have negative correlations or values (or ignore the signs?)\n\n5. Fig 5: Some symbols were lost.", "summary_of_the_paper": "The authors proposed a method to recover time-delayed latent causal variables and identify their relations from measured temporal data called Latent tEmporally cAusal Processes estimation (LEAP). This method leverages VAEs by enforcing our conditions through proper constraints in the causal process prior for recovering time-delayed latent processes from nonlinear mixtures without sparsity assumptions. The experimental results on synthetic datasets show that the proposed method outperformed baselines that do not leverage history or nonstationarity information. The results on public datasets including KiTTiMask, Mass-Spring System, and CMU MoCap demonstrate that temporally causal latent processes were reliably identiﬁed from observed variables under different dependency structures.\n", "summary_of_the_review": "The problem setting seems to be completely new, experiments were sufficiently performed, the results show the superiority of the proposed method, whereas the technical contributions are derived from mixing the previous techniques. \nSince I think the strengths in this paper totally outperformed the weaknesses, I provided a higher rating at this stage (but note that I am not an expert in this field). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes two new sets of conditions under which we can identify temporally causal latent processes. In this sense, this work makes valuable contributions to the theories of identifiability in this topic. The authors also propose LEAP, extending the VAE, to estimate temporally causal latent processes.

The reviewers had many constructive comments, and the authors strived to address them. In the end, the reviewers were satisfied with the final version of the paper. 

Given that the theoretical identifiability theorems are major parts of the paper, I encourage the authors to elaborate more on the two sets of assumptions. They should discuss when these assumptions will hold and provide examples in which they will be violated.
Decision: Accept (Poster)