Learning Temporally Latent Causal Processes from General Temporal Data

1 INTRODUCTION AND RELATED WORK . Causal discovery seeks to identify the underlying structure of the data generation process by exploiting an appropriate class of assumptions ( Spirtes et al. , 1993 ; Pearl , 2000 ) . Despite its success in certain domains , most existing work either focuses on estimating the causal relations between observed variables ( Spirtes & Glymour , 1991 ; Chickering , 2002 ; Shimizu et al. , 2006 ) , or starts from the premise that causal variables are given beforehand ( Spirtes et al. , 2013 ) . Real-world observations ( e.g. , image pixels , sensor measurements , etc . ) , however , are not structured into causal variables to begin with . Estimating latent causal variable graphs from observations is particularly challenging as the latent variables , even with independent factors of variation ( Locatello et al. , 2019 ) , are not identifiable or “ uniquely ” recoverable in the most general case ( Hyvärinen & Pajunen , 1999 ) . There exist several pieces of work aiming to uncover causally related latent variables . For instance , by exploiting the vanishing Tetrad conditions ( Spearman , 1928 ) one is able to identify latent variables in linear-Gaussian models ( Silva et al. , 2006 ) , and the so-called Generalized Independent Noise ( GIN ) condition was proposed to estimate linear , non-Gaussian latent variable causal graph ( Xie et al. , 2020 ) , with follow-up studies such as ( Adams et al. , 2021 ) . However , these approaches are constrained within linear relations , need certain types of sparsity or minimality assumptions , and require a relatively large number of measured variables as children of the latent variables . The work of ( Bengio et al. , 2019 ; Ke et al. , 2019 ) used “ quick adaptation ” as the training criteria for learning latent structure but the identifiability results have not been theoretically established yet . Recent advances in the theory of nonlinear Independent Component Analysis ( ICA ) have proven strong identifiability results ( Hyvarinen & Morioka , 2016 ; 2017 ; Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020 ; Sorrenson et al. , 2020 ) by exploiting certain side information in addition to independence . By assuming that the generative latent factors zi are conditionally independent ∗ Equal contribution . Code : https : //github.com/weirayao/leap given auxiliary variables u that may be time index , domain index , class label , etc . and augmenting observation data x with u , deep generative models fit with such tuples ( x , u ) may be identifiable in function space ; they can recover independent factors up to a certain transformation of the original latent variables under proper assumptions ( note that we use “ latent factor ” and “ latent processe ” interchangeably ) . Although the temporal structure is widely used for nonlinear ICA , existing work that establishes identifiability results considers only independent sources , or further with linear transitions . However , these assumptions may severely distort the results if the real latent factors have causal relations in between , or if the relations are nonlinear . It is not clear yet how the temporal structure may help in learning temporally causally-related latent factors , together with their causal structure , from temporal observation data . In this paper , we focus on the scenario where the observed temporal variables xt do not have direct causal edges but are generated by latent processes or confounders zt that have time-delayed causal relations in between . That is , the observed data xt are unknown nonlinear ( but invertible ) mixtures of the underlying sources : xt = g ( zt ) . Our first goal is hence to understand under what conditions the latent temporally causal processes can be identified . Inspired by real situations , we consider both a nonparametric , nonstationary setting and a parametric setting for the latent processes . In the nonparametric setting , the generating process of each latent causal factor zit is characterized by nonparametric assignment zit = fi ( Pa ( zit ) , ϵit ) , in which the parents of zit ( i.e. , the set of latent factors that directly cause zit ) together with noise term ϵit ∼ pϵ ( where pϵ denotes the distribution of ϵit ) generate zit via unknown nonparametric function fi with some time delay . In the parametric setting , the time-delayed causal influences among latent factors follow a linear form . In both settings , we establish the identifiability of the latent factors and their causal influences , rendering them recoverable from the observed data . Our second goal is then to develop a theoretically-grounded training framework that enforces the assumed conditions through proper constraints . To this end , we propose Latent tEmporally cAusal Processes estimation ( LEAP ) , a novel architecture that extends VAEs with a learned causal process prior network that enforces the Independent Noise ( IN ) condition and models possible nonstationarity through flow-based estimators . We evaluate LEAP on a number of synthetic and real-world datasets , including video and motion capture data with the properties required by our conditions . Experimental results demonstrate that temporally causal latent processes are reliably identified from observed variables under different dependency structures , and that our approach considerably outperforms existing methods which do not leverage history or nonstationarity information . The closest work to ours includes ( Klindt et al. , 2020 ; Hyvarinen & Morioka , 2017 ) , which require the underlying sources to be mutually independent for identifiability . Our work extends the theories for the discovery of conditionally independent sources to the setting with temporally causally-related latent processes , by leveraging nonstationarity or functional and distributional constraint on temporal relations . To the best of our knowledge , this is one of the first works that successfully recover time-delayed latent processes from their nonlinear mixtures without using sparsity or minimality assumptions . The proposed framework may serve as an alternative tool for creating versatile models robust to domain shifts and may be extended for more general conditions with changing causal relations over time . 2 IDENTIFIABILITY THEORY . 2.1 IDENTIFIABILITY PROPERTY . We summarize the recent literature related to our work from four perspectives and compare our proposed theory with them in Table 1 . The detailed comparisons of the problem settings are given in Appendix A.4 . Following prior work , we define identifiability in representation function space . Definition 1 ( Componentwise Identifiability ) Let xt be a sequence of observed variables generated by the true temporally causal latent processes specified by ( fi , pϵi ) and nonlinear mixing function g , given in the introduction . A learned generative model ( ĝ , f̂i , p̂ϵi ) is observationally equivalent to ( g , fi , pϵi ) if the joint distribution pĝ , f̂ , p̂ϵ ( xt ) matches pg , f , pϵ ( xt ) everywhere . We say latent causal processes are identifiable if observational equivalence can always lead to identifiability of the latent variables up to permutation π and component-wise invertible transformation T : pĝ , f̂ , p̂ϵ ( xt ) = pg , f , pϵ ( xt ) ⇒ ĝ = g ◦ T ◦ π . ( 1 ) Once the latent causal processes are identifiable up to componentwise transformations , latent causal relations are also identifiable because conditional independence relations fully characterize timedelayed causal relations in a time-delayed causally sufficient system . Note that invertible componentwise transformations on latent causal processes do not change their conditional independence relations . 2.2 OUR PROPOSED CONDITIONS . We consider two novel conditions that ensure the identifiability of temporally causal latent processes using ( 1 ) nonstationarity or ( 2 ) functional and distributional constraints on their temporal relations . The corresponding identifiability of the latent processes is established in the following two theorems , with proofs and discussions of the assumed conditions provided in Appendix A. Theorem 1 ( Nonparametric Processes ) Assume nonparametric processes in Eq . 2 , where the transition functions fi are third-order differentiable functions and mixing function g is injective and differentiable almost everywhere ; let Pa ( zit ) denote the set of ( time-delayed ) parent nodes of zit : xt = g ( zt ) ︸ ︷︷ ︸ Nonlinear mixing , zit = fi ( { zj , t−τ |zj , t−τ ∈ Pa ( zit ) } , ϵit ) ︸ ︷︷ ︸ Nonparametric transition with ϵit ∼ pϵi|u︸ ︷︷ ︸ Nonstationary noise . ( 2 ) Here we assume : 1 . ( Nonstationary Noise ) : Noise distribution pϵi|u is modulated ( in any way ) by the observed cate- gorical auxiliary variables u , which denotes nonstationary regimes or domain index ; 2 . ( Independent Noise ) : The noise terms ϵit are mutually independent ( i.e. , spatially and temporally independent ) in each regime of u ( note that this directly implies that ϵit are independent from . Pa ( zit ) in each regime ) ; 3 . ( Sufficient Variability ) : For any zt ∈ Rn there exist 2n + 1 values for u , i.e. , uj with . j = 0 , 1 , ... , 2n , such that the 2n vectors w ( zt , uj+1 ) − w ( zt , uj ) , with j = 0 , 1 , ... , 2n , are linearly independent with w ( zt , u ) defined below , where qi is the log density of the conditional distribution and zHx = { zt−τ } denotes history information up to maximum time lag L : w ( zt , u ) ≜ ( ∂q1 ( z1t|zHx , u ) ∂z1t , . . . , ∂qn ( znt|zHx , u ) ∂znt , ∂2q1 ( z1t|zHx , u ) ∂z21t , . . . , ∂2qn ( znt|zHx , u ) ∂z2nt ) . ( 3 ) Then the componentwise identifiability property of temporally causal latent processes is ensured . Theorem 2 ( Parametric Processes ) Assume the vector autoregressive process in Eq . 4 , where the state transition functions are linear and additive and mixing function g is injective and differentiable almost everywhere . Let Bτ ∈ Rn×n be the state transition matrix at lag τ . The process noises ϵit … 𝐻 ! C. Causal Process ? ̂ ? `` , $ ? ̂ ? % , $ … … … ? ̂ ? `` , $ & ' ? ̂ ? % , $ & ' … … … ? ̂ ? '' , $ & ! ? ̂ ? % , $ & ! … … … … … … Time-delayed Information Element / Channel Index Transition Prior Modeling 𝑝 ( 𝐳 $ 𝒕| 𝐳 $ 𝒕 '' 𝝉 $ % & ' ) Time Index … … 𝐻 ( 𝐻' … … … 𝐻 ! 𝐻 ( 𝐻' 𝐳 % 𝑳 … …𝐱 % 𝑳 …𝐱𝐋… 𝐱 % 𝑳 B . A. Enc . A. Enc . A. Enc . 𝐱𝟏 𝐱𝑻 𝐱 % 𝟏 𝐳 % 𝟏 𝐳 % 𝑻 D. Dec. D. Dec. D. Dec . Figure 2 : LEAP : Encoder ( A ) and Decoder ( D ) with MLP or CNN for specific data types ; ( B ) Bidirectional inference network that approximates the posteriors of latent variables ẑ1 : T , and ( C ) Causal process network that ( 1 ) models nonstationary latent causal processes ẑt with Independent Noise constraint ( Thm 1 ) or ( 2 ) models the linear transition matrix with Laplacian constraints ( Thm 2 ) . are assumed to be stationary and both spatially and temporally independent : xt = g ( zt ) ︸ ︷︷ ︸ Nonlinear mixing , zt = L∑ τ=1 Bτzt−τ + ϵt︸ ︷︷ ︸ Linear additive transition with ϵit ∼ pϵi︸ ︷︷ ︸ Independent noise . ( 4 ) Here we assume :

Summary: The paper considers the learning of a temporal latent causal process: x_t=g(z_t), where x_t are observations, z_t are the underlying latent variables, and g is unknown. The paper makes two contributions: 1) presenting conditions under which the process can be identified (up to a permutation of the latent variables and their componentwise invertible transformations), and 2) developing a training framework that enforces the assumed conditions. The first contribution is theoretical and the second methodological. In addition, the paper presents empirical evidence for the ability of the method to recover the underlying causal process using synthetic experiments and real-world data.  Two cases are considered: 1) non-parametric, where a non-parametric transition is assumed in the latent space, and 2) parametric, where linear additive transitions are assumed. The main assumption for identifiability is the independent noise (IN) condition, which states that noise for the dimensions of z_t are independent of each other and across time steps t. This assumption is enforced through a contrastive approach during training. In addition, in the non-parametric case the noise is assumed non-stationary, such there exists sufficiently many auxiliary variables that affect the distribution of the non-stationary noise.