Causal Temporal Representation Learning with Nonstationary Sparse Transition

Authors: xiangchen  song
zijian  li
guangyi  chen
yujia  zheng
yewen  fan
xinshuai  dong
kun  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 23:17:35

Abstract: Causal Temporal Representation Learning (Ctrl) methods aim to identify the temporal causal dynamics of complex nonstationary temporal sequences. Despite the success of existing Ctrl methods, they require either directly observing the domain variables or assuming a Markov prior on them. Such requirements limit the application of these methods in real-world scenarios when we do not have such prior knowledge of the domain variables. To address this problem, this work adopts a sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. In particular, we explore under what conditions on the significance of the variability of the transitions we can build a model to identify the distribution shifts. Based on the theoretical result, we introduce a novel framework, *Causal Temporal Representation Learning with Nonstationary Sparse Transition* (CtrlNS), designed to leverage the constraints on transition sparsity and conditional independence to reliably identify both distribution shifts and latent factors. Our experimental evaluations on synthetic and real-world datasets demonstrate significant improvements over existing baselines, highlighting the effectiveness of our approach.

Main paper: 
 introduction: 
 Causal learning from sequential data remains a fundamental yet challenging task [1] [2] [3] . Discovering temporal causal relations among observed variables has been extensively studied in the literature [4] [5] [6] . However, in many real-world scenarios such as video understanding [7] , observed data are generated by causally related latent temporal processes or confounders rather than direct causal edges. This leads to the task of causal temporal representation learning (Ctrl), which aims to build compact representations that concisely capture the data generation processes by inverting the mixing function that transforms latent factors into observations and identifying the transitions that govern the underlying latent causal dynamics. This learning problem is known to be challenging without specific assumptions [8, 9] . The task becomes significantly more complex with nonstationary transitions, which are often characterized by multiple distribution shifts across different domains, particularly when these domains or shifts are also unobserved. We first introduce a nonstationary time-series generative model in our setting. The observational dataset is D = {x t } T t=1 , where x t ∈ R n is produced from causally related, time-delayed latent components z t ∈ R n through an invertible mixing function g:EQUATIONIn the nonstationary setting, transitions within the latent space vary over time. Define u as the domain or regime index variable, with u t corresponding to domain variable at time step t. Assuming there are U distinct regimes, i.e., u t ∈ {1, 2, . . . , U }, each regime exhibits unknown distribution shifts. Those regimes are characterized by U different transition functions {m u } U u=1 , which were originally introduced in [32] through change factors to capture these distribution shifts in transition dynamics. The i-th component of latent variable z t , is then generated via i-th component of transition function m:EQUATIONwhere Pa(z t,i ) represents the set of latent factors directly influencing z t,i , which may include any subset of z <t = {z τ,i | τ ∈ {1, 2, . . . , t -1}, i ∈ {1, 2, . . . , n}}. For analytical simplicity, we assume that the parents in the causal graph are restricted to elements in z t-1 . Extensions to higherorder cases, which involve multistep, time-delayed causal relations, are already discussed in Appendix S1.5 of [23] . These extensions are orthogonal to our contributions and are therefore omitted here for brevity. Importantly, in a nonstationary context, Pa(•) may also be sensitive to the domain index u t , indicating that causal dependency graphs vary across different domains or regimes, which will be revisited in our later discussion on identifiability. We assume that the generation processes for each i-th component of z t are mutually independent, given z <t and u t . Consistent with the existing literature [23, 25] , we further assume that the noise terms ϵ t,i are independent both spatially and temporally. This assumption implies that there is no instantaneous causal influence among the latent causal processes. The graphical model corresponding to this setting is illustrated in Figure 1 . 
identifiability of domain variables and latent causal processes: 
 In this section we introduce the identifiability of both domain variables and time-delayed latent causal processes in Definitions 2 and 3, respectively. If the estimated latent processes are identifiable at least up to a permutation and component-wise invertible transformations, then the latent causal relationships are also immediately identifiable. This follows from the fact that conditional independence relations comprehensively characterize the time-delayed causal relations within a time-delayed causally sufficient system, in which there are no latent causal confounders in the causal processes. Notably, invertible component-wise transformations on latent causal processes preserve their conditional independence relationships. We now present definitions concerning observational equivalence, the identifiability of domain variables and latent causal processes. Definition 1 (Observational Equivalence). Formally, consider {x t } T t=1 as a sequence of observed variables generated by true temporally causal latent processes specified by (m, u, p(ϵ), g) given in Eqs. (1) and (2) . Here, m and ϵ denote the concatenated vector form across n dimensions in the latent space. Similarly u for time steps 1 to T . A learned generative model ( m, û, p(ϵ), ĝ) is observationally equivalent to the ground truth one (m, u, p(ϵ), g) if the model distribution p m,û, pϵ, ĝ({x t } T t=1 ) matches the data distribution p m,u,pϵ,g ({x t } T t=1 ) everywhere. Definition 2 (Identifiable Domain Variables). Domain variables are said to be identifiable up to label swapping if observational equivalence (Def. 1) implies identifiability of domain variables up to a permutation σ for domain indices: p m,û, pϵ, ĝ({x t } T t=1 ) = p m,u,pϵ,g ({x t } T t=1 ) ⇒ ût = σ(u t ), ∀t ∈ {1, 2, . . . , T }.Definition 3 (Identifiable Latent Causal Processes). The latent causal processes are said to be identifiable if observational equivalence (Def. 1) leads to the identifiability of latent variables up to a permutation π and component-wise invertible transformation T : p m,û, pϵ, ĝ({x t } T t=1 ) = p m,u,pϵ,g ({x t } T t=1 ) ⇒ ĝ-1 (x t ) = T • π • g -1 (x t ), ∀x t ∈ X , (4) where X denotes the observation space. 
identifiability theory: 
 In this section, we demonstrate that under mild conditions, the domain variables u t are identifiable up to label swapping and the latent variables z t are identifiable up to permutation and component-wise transformations. We partition our theoretical discussion into two sections: (1) identifiability of nonstationary discrete domain variables u t and (2) identifiability of latent causal processes. We slightly extend the usage of supp(•) to define the square matrix support and the support of a square matrix function as follows: Definition 4 (Matrix Support). The support (set) of a square matrix A ∈ R n×n is defined using the indices of non-zero entries as:EQUATION)Definition 5 (Matrix Function Support). The support (set) of a square matrix function A : Θ → R n×n is defined as: supp(A(Θ)) := {(i, j) | ∃θ ∈ Θ, A(θ) i,j ̸ = 0} .For brevity, let M and M denote the n × n binary matrices representing the support of the Jacobian J m (z t ) and J m(ẑ t ), respectively. The (i, j)-th entry of M is 1 if and only if (i, j) ∈ supp(J m ). We further define the transition complexity using its Fréchet norm as |M| = i,j M i,j , and similarly for M. In the nonstationary setting, this support matrix becomes a function of the domain index u, denoted as M u and M u . Additionally, we introduce the concept of weakly diverse lossy transitions for the data generation process, which is formally defined below:Definition 6 (Weakly Diverse Lossy Transition). The set of transition functions described in Eq. ( 2) is said to be diverse lossy if it satisfies the following conditions:1. (Lossy) For every time and indices tuple (t, i, j) with edge z t-1,i → z t,j representing a causal link defined with the parents set Pa(z t,j ) in Eq. 2, transition function m j is a lossy transformation w.r.t. z t-1,i i.e., there exists an open set S t,i,j1 , changing z t-1,i within this set will not change the value of m j , i.e. ∀z t-1,i ∈ S t,i,j , ∂mj ∂zt-1,i = 0. 2. (Weakly Diverse) For every element z t-1,i of the latent variable z t-1 and its corresponding children set J t,i = {j | z t-1,i ∈ Pa(z t,j ), j ∈ {1, 2, . . . , n}}, transition functions {m j } j∈Jt,i are weakly diverse i.e., the intersection of the sets S t,i = ∩ j∈Jt,i S t,i,j is not empty, and such sets are diverse, i.e., S t,i ̸ = ∅, and S t,i,j \ S t,i ̸ = ∅, ∀j ∈ J t,i .Theorem 1 (Identifiability of Domain Variables). Suppose that the dataset D are generated from the nonstationary data generation process as described in Eqs.(1) and (2) . Suppose the transitions are weakly diverse lossy (Def. 6) and the following assumptions hold:i. (Mechanism Separability) There exists a ground truth mapping C : X × X → U determined the real domain indices, i.e., u t = C(x t-1 , x t ).ii. (Mechanism Sparsity) The estimated transition complexity on dataset D is less than or equal to ground truth transition complexity, i.e.,E D | M û| ≤ E D |M u |.iii. (Mechanism Variability) Mechanisms are sufficiently different. For all u ̸ = u ′ , M u ̸ = M u ′ i.e. there exists index (i, j) such that[M u ] i,j ̸ = [M u ′ ] i,j .Then the domain variables u t is identifiable up to label swapping (Def. 2).Theorem 1 states that if we successfully learn a set of estimated transitions { mu } U u=1 , the decoder ĝ, and the domain clustering assignment Ĉ, where mu corresponds to the estimation of Eq. (2) for a particular regime or domain u, and the system can fit the data as follows:EQUATIONassuming that the transition complexity is kept low (as per Assumption ii). Then the estimated domain variables ût must be the true domain variables u t up to a permutation.The core idea of this proof is to demonstrate that the global minimum of transition complexity can only be achieved when the domain variables u t are correctly estimated. (1) First, we consider the case when we have an optimal decoder estimation ĝ * which is a component-wise transformation of the ground truth, incorrect estimations of u t will strictly increase the transition complexity, i.e.,E D | M * û| > E D | M * u |.(2) Second, we show that with arbitrary estimations ût , the transition complexity for any non-optimal decoder estimation ĝ will be equal to or higher than that for the optimal ĝ * , i.e.,E D | M û| ≥ E D | M * û|.Thus, the global minimum of transition complexity can only be achieved when u t is optimally estimated, which must be a permuted version of the ground truth domain variables u t . A comprehensive proof can be found in Appendix S1.1.The assumption of mechanism variability, as stated in Assumption iii, requires that the Jacobian support matrices differ across domains, indicating that the causal graph connecting past states (z t-1 ) to current states (z t ) must differ by at least one edge. Addressing scenarios where the causal graphs remain identical but the transition functions associated with the edges vary is generally challenging without imposing additional assumptions. A more detailed discussion of the difficulties involved in such cases is provided in Appendix S1.4.4. To effectively address these scenarios, we extend the concept of the Jacobian support matrix by incorporating higher-order derivatives. This extension provides a more detailed characterization of the variability in transition functions across different domains. We now present the following definition to formalize this concept: Definition 7 (Higher Order Partial Derivative Support Matrix). The k-th order partial derivative support matrix for transition m denoted as M k is a binary n × n matrix withEQUATIONWe utilize the variability in the higher-order partial derivative support matrix to extend the identifiability results of Theorem 1. This extension applies to cases where the causal graphs remain identical across two domains, yet the transition functions take different forms.Corollary 1 (Identifiability under Function Variability). Suppose the data D is generated from the nonstationary data generation process described in (1) and (2). Assume the transitions are weakly diverse lossy (Def. 6), and the mechanism separability assumption i along with the following assumptions hold:v. (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K ∈ N such that for all u ̸ = u ′ , there existsk ≤ K, M k u ̸ = M k u ′ i.e. there exists index (i, j) such that M k u i,j ̸ = M k u ′ i,j .vi. (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset D is no more than ground truth transition complexity,EQUATION)Then the domain variables u t are identifiable up to label swapping (Def. 2).To prove this corollary, we leverage the property that, for any two distinct domains, there exists an edge in the causal graph such that the supports of their k-th order partial derivatives differ. This difference ensures the separability of the two domains. A detailed proof can be found in Appendix S1.2.Once the identifiability of u t is achieved, the problem reduces to a nonstationary temporal nonlinear ICA with observed domain index. Leveraging the sufficient variability approach proposed in [23] , we demonstrate full identifiability of the data generation process. This sufficient variability concept is further incorporated into the following lemma, adapted from Theorem 2 in [23] : Lemma 1 (Theorem 2 in Yao et al., [23] ). Suppose that the data D are generated from the nonstationary data generation process as described in Eqs. (1) and (2). Let η kt (u) denote the logarithmic density of k-th variable in z t , i.e., η kt (u) ≜ log p(z t,k |z t-1 , u), and there exists an invertible function ĝ that maps x t to ẑt , i.e., ẑt = ĝ(x t ) such that the components of ẑt are mutually independent conditional on ẑt-1 . (Sufficient variability) LetEQUATION)EQUATION)EQUATIONEQUATION)Suppose x t = g(z t ) and that the conditional distribution p(z k,t | z t-1 ) may change across m domains. Suppose that the components of z t are mutually independent conditional on z t-1 in each context. Assume that the components of ẑt produced by ĝ are also mutually independent conditional on ẑt-1 . If the 2n function vectors s k,t and sk,t , with k = 1, 2, ..., n, are linearly independent, then ẑt is a permuted invertible component-wise transformation of z t .Then, in conjunction with Theorem 1, complete identifiability is achieved for both the domain variables u t and the independent components z t . See detailed proof in Appendix S1.3.Theorem 2 (Identifiability of the Latent Causal Processes). Suppose that the observed dataset D is generated from the nonstationary data generation process as described in Eqs. (1) and (2), which satisfies the conditions in both Theorem 1 and Lemma 1, then the domain variables u t are identifiable up to label swapping (Def. 2) and latent causal process z t are identifiable up to permutation and a component-wise transformation (Def. 3).The proof of Theorem 1 relies on several essential assumptions that correspond with human intuition regarding domain transitions. First, the assumption of separability posits that if human observers are unable to differentiate between two domains, it is improbable that automated systems will achieve such a distinction. Second, the variability assumption requires that the differences in transitions between domains be substantial enough to be perceptible to humans. This often results in changes to the temporal causal structure across domains, indicating that at least one edge in the causal graph must differ between the domains.The mechanism sparsity is a standard assumption that has been previously explored in [33, 19, 18] using sparsity regularization to enforce the sparsity of the estimated function. The assumption of weakly diverse lossy transitions is a mild and realistic condition in real-world scenarios, allowing for identical future latent states with differing past states. The sufficient variability in Theorem 2 is widely explored and adopted in nonlinear ICA literature [12, 22, 23, 25, 26] . For a more detailed discussion of the feasibility and intuition behind these assumptions, we refer the reader to the Appendix S1.4.4 The CtrlNS Framework i I N N m E 1 B N 7 u m 6 9 d x N C t V N D E 0 c I 8 = " > A A A C H H i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 V w I S W p U l 0 W 3 b i s 0 B c 0 I U y m k 3 b o Z B J m J m I J / R A 3 / o o b F 4 q 4 c S H 4 N 0 7 a L P r w w M C 5 5 z 7 m 3 u P H j E p l W b 9 G Y W 1 9 Y 3 O r u F 3 a 2 d 3 b P z A P j 9 o y S g Q m L R y x S H R 9 J A m j n L Q U V Y x 0 Y 0 F Q 6 D P S 8 U d 3 W b 7 z S I S k E W + q c U z c E A 0 4 D S h G S k u e e e m k T o j U 0 A / S p 4 l n X 8 C 5 q K q j f q T k g t h 0 J p 5 Z t i r W F H C V 2 D k p g x w N z / z W c 3 A S E q 4 w Q 1 L 2 b C t W b o q E o p i R S c l J J I k R H q E B 6 W n K U U i k m 0 6 P m 8 A z r f R h E A n 9 u I J T d b 4 j R a G U 4 9 D X l d m W c j m X i f / l e o k K b t y U 8 j h R h O P Z R 0 H C o I p g 5 h T s U 0 G w Y m N N E B Z U 7 w r x EA m E l f a z p E 2 w l 0 9 e J e 1 q x a 5 V a g 9 X 5 f p t b k c R n I B T c A 5 s c A 3 q 4 B 4 0 Q A t g 8 A x e w T v 4 M F 6 M N + P T + J q V F o y 8 5 x g s w P j 5 A / E U o e w = < / l a t e x i t > + C w p c J Y U t a k o Q h l x y W K C R 6 w J n A Q r B N J R n x X s L Y 7 v s n 8 9 g O T i o d B A y Y R 6 / l k G H C P U w J a 6 p u 3 T u K M C C S O T 2 D k e s l j m v b t c 7 y g V b Q 2 C E E t s R p O 2 j d L V t m a A i 8 S O y c l l K P e N 1 / 1 N B r 7 L A A q i F J d 2 4 q g l x A J n A q W F p 1 Y s Y j Q M R m y r q Y B 8 Z n q J d N z U 3 y q l Q H 2 Q q l f A H i q / u 1 I i K / U x H d 1 Z b a m m v c y c Z n X j c G 7 6 i U 8 i G J g A Z 1 9 5 M U C Q 4 i z 7 P C A S 0 Z B T D Q h V H K 9 K 6 Y j I g k F n X B R h 2 D P n 7x I W p W y X S 1 X 7 y 9 K t e s 8 j g I 6 R i f o D N n o E t X Q H a q j J q L o C b 2 g D / R p P B t v x p f x P S t d M f K e I / Q P x s 8 v 8 q K q U w = = < / l a t e x i t > {x 1 , x2 , . . . , xT } < l a t e x i t s h a 1 _ b a s e 6 4 = " h 9 c 4 s i l 3 o N g s g p O V I q B T Y e 0 2 Y S s = " > A A A C A 3 i c b V A 9 S w N B E N 3 z M 8 a v q J 0 2 h 0 G w C n c i 0 T J o Y 2 E R x X x A E s L e Z i 5 Z s r d 3 7 M 6 J 4 T i w 8 a / Y W C h i 6 5 + w 8 9 + 4 l 6 T Q x A c D j / d m m J n n R Y J r d J x v a 2 F x a X l l N b e W X 9 / Y 3 N o u 7 O z W d R g r B j U W i l A 1 P a p B c A k 1 5 C i g G S m g g S e g 4 Q 0 v M 7 9 x D 0 r z U N 7 h K I J O Q P u S + 5 x R N F K 3 s N 8 O K A 4 Y F c l 1 2 k 3 a C A + Y 3 A I L Z Z p 2 C 0 W n 5 I x h z x N 3 S o p k i m q 3 8 N X u h S w O Q C I T V O u W 6 0 T Y S a h C z g S k + X a s I a J s S P v Q M l T S A H Q n G f + Q 2 k d G 6 d l + q E x J t M f q 7 4 m E B l q P A s 9 0 Z h f r W S 8 T / / N a M f r n n Y T L K E a Q b L L I j 4 W N o Z 0 F Y v e 4 A o Z i Z A h l iQ = " > A A A B 9 X i c b V B N S 8 N A E N 3 4 W e t X 1 a O X Y B E 8 l U S k e i y K 4 L G C / Y A 2 l s 1 m 0 i 7 d b M L u R C 2 h / 8 O L B 0 W 8 + l + 8 + W / c t j l o 6 4 O B x 3 s z z M z z E 8 E 1 O s 6 3 t b S 8 s r q 2 X t g o b m 5 t 7 + y W 9 v a b O k 4 V g w a L R a z a P t U g u I Q G c h T Q T h T Q y B f Q 8 o d X E 7 / 1 A E r z W N 7 h K A E v o n 3 J Q 8 4 o G u m + i / C E 2 b V k c Q B q 3 C u V n Y o z h b 1 I 3 J y U S Y 5 6 r / T V D W K W R i C R C a p 1 x 3 U S 9 D K q k D M B 4 2 I 3 1 Z B Q N q R 9 6 B g q a Q T a y 6 Z X j + 1 j o w R 2 G C t T E u 2 p + n s i o 5 H W o 8 g 3 n R H F g Z 7 3 J u J / X i f F 8 M L L u E x S B M l m i 8 J U 2 B j b k w j s g C t g K E a G U K a 4 u d V m A 6 o o Q x N U 0 Y T g z r + 8 S J q n F b d a q d 6 e l W u X e R w F c k i O y A l x y T m p k R t S J w 3 C i C L P 5 J W 8 W Y / W i / V u f c x a l 6 x 8 5 o D 8 g f X 5 A x c 9 k u o = < / l a t e x i t > Encoder < l a t e x i t s h a _ b a s e = " V e j S V F t J b v J w J O D V s M A N T t = " > A A A B X i c b V B N S N A E N W e t X a O X Y B E l U S k e i z q w W M F + w F t L J v N p F y Y b d i V p C / c X D p b + b X P Q g c D j / d m m J n n J J r d J x v a l Z X V t v b B R N z a t k t e t U w V g w a T Q q q T z U I H k M D O Q p o J w p o A t o + c O r i d A K W j O w l I A X X M Q o G u m + i / C E T U w G Y A a p l p + J M Y S S N y d l k q P e K A n S C G J k g m r d c Z E v Y w q E z A u N h N N S S U D W k f O o b G N A L t Z d O r x / a x U Q I l M p U j P Z U / T R U j r U e S b z o j i Q M E / E / r N i e O F l P E S h J j N F o W p s F H a k w j s g C t g K E a G U K a u d V m A o o Q x N U Y T g z r + S J q n F b d a q d e l W u X e R w F c k i O y A l x y T m p k R t S J w C i C L P J W W Y / W i / V u f c x a l x o D g f X A w f Y k u A = < /q i F J d 2 4 q g l x A J n A q W F p 1 Y s Y j Q M R m y r q Y B 8 Z n q J d N z U 3 y q l Q H 2 Q q l f A H i q / u 1 I i K / U x H d 1 Z b a m m v c y c Z n X j c G 7 6 i U 8 i G J g A Z 1 9 5 M U C Q 4 i z 7 P C A S 0 Z B T D Q h V H K 9 K 6 Y j I g k F n X B R h 2 D P n 7x I W p W y X S 1 X 7 y 9 K t e s 8 j g I 6 R i f o D N n o E t X Q H a q j J q L o C b 2 g D / R p P B t v x p f x P S t d M f K e I / Q P x s 8 v / H K q W Q = = < / l a t e x i t > M g R Q Q V F C i h H m t g o S + h 5 v c v R 3 7 t D r Q R K r r F Q Q y t k H U j E Q j O 0 E r t 3 H 4 T 4 Q H T s h Z K 0 2 v A e 6 X 7 w 3 Y u 7 x b c M e g 8 8 a Y k T 6 Y o t 3 N f z Y 7 i S Q g R c s m M a X h u j K 2 U a R R c w j D b T A z E j P d Z F x q W R i w E 0 0 r H 6 Y f 0 y C o d G t j 7 g Y q Q j t X f G y k L j R m E v p 0 M G f b M r D c S / / M a C Q b n r V R E c Y I Q 8 c m h I J E U F R 1 V Q T t C A 0 c 5 s I R x L Wx W y n t M M 4 6 2 s K w t w Z t 9 e Z 5 U T w p e s V C 8 O c 2 X L q Z 1 Z M g B O S T H x C N n p E S u S J l U C C e P 5 J m 8 k j f n y X l x 3 p 2 P y e i C M 9 3 Z I 3 / g f P 4 A H 9 + V r Q = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " E f P J d S I X k Q M t q C 1 H E i m f g F U + g 3 A = " > A A A C B H i c b V A 9 S w N B E N 3 z M 8 a v U 8 s 0 i 0 G w C n c i 0 T J o Y 2 E R 0 X x A E s L e Z p I s 2 f t g d 0 4 M x x U 2 / h U b C 0 V s / R F 2 / h v 3 k h S a + G D g 8 d 4 M M / O 8 S A q N j v N t L S 2 v r K 6 t 5 z b y m 1 v b O 7 v 2 3 n 5 d h 7 H i U O O h D F X T Y x q k C K C G A i U 0 I w X M 9 y Q 0 v N F l 5 j f u Q W k R B n c 4 j q D j s 0 E g + o I z N F L X L r Ri Q t P 1 y t O X E = " > A A A C B H i c b V C 7 T s M w F H X K q 5 R X g L G L R Y X E V C U I F c Y K F s Y i 0 Y f U R J X j O K 1 V x 4 5 s B 6 m K M r D w K y w M I M T K R 7 D x N z h t B m g 5 k u W j c + 7 V v f c E C a N K O 8 6 3 V V l b 3 9 j c q m 7 X d n b 3 9 g / s w 6 O e E q n E p I s F E 3 I Q I E U Y 5 a S r q W Z k k E i C 4 o C R f j C 9 K f z + A 5 G K C n 6 v Z w n x Y z T m N K I Y a S O N 7 L o 3 Q T r z A s F C N Y v N l 3 k k U Z Q J n u c j u + E 0 n T n g K n F L 0 g A l O i P 7 y w s F T m P C N W Z I q a H r J N r P k N Q U M 5 L X v F S R B O E p G p O h o R z F R P n Z / I g c n h o l h J G Q 5 n E N 5 + r v j g z F q l j R V M Z I T 9 S y V 4 j / e c N U R 1 d + R n m S a s L x Y l C U M q g F L B K B I Z U E a z Y z B G F J z a 4 Q T 5 B E W J v c a i Y E d / n k V d I 7 b 7 q t Z u v u o t G + L u O o g j o 4 A W f A B Z e g D W 5 B B 3 Q B B o / g G b y C N + v J er H e r Y 9 F a c U q e 4 7 B H 1 i f P 1 q B m T Y = < / l a t e x i t > ✏ < l a t e x i t s h a 1 _ b a s e 6 4 = " g YS k l E V N 4 Q Y o 4 8 s I h a + m Q o n V z Y A = " > A A A B / n i c b V D L S s N A F L 3 x W e s r K q 7 c B I v g q i Q i 1 W X R j c s K 9 g F N K J P J p B 0 6 m Q k z E 6 G E g r / i x o U i b v 0 O d / 6 N k z Y L b T 0 w z O G c e 5 k z J 0 w Z V d p 1 v 6 2 V 1 b X 1 j c 3 K V n V 7 Z 3 d v 3 z 4 4 7 C i R S U z a W D A h e y F S h F F O 2 p p q R n q p J C g J G e m G 4 9 v C 7 z 4 S q a j g D 3 q S k i B B Q 0 5 j i p E 2 0 s A + 9 k P B I j V J z J X 7 J F W U C T 4 d 2 D W 3 7 s 7 g L B O v J D U o 0 R r Y X 3 4 k c J Y Q r j F D S v U 9 N 9 V B j q S m m J F p 1 c 8 U S R E e o y H p G 8 p R Q l S Q z + J P n T O j R E 4 s p D l c O z P 1 9 0 a O E l U k N J M J 0 i O 1 6 B X i f 1 4 / 0 / F 1 k F O e Z p p w P H 8 o z p i j h V N 0 4 U R U E q z Z x B C E J T V Z H T x C E m F t G q u a E r z F L y + T z k X d a 9 Q b 9 5 e 1 5 k 1 Z R w V O 4 B T O w Y M r a M I d t K A N G H J 4 h l d 4 s 5 6 s F + v d + p i P r l j l z h H 8 g f X 5 A 1 z b l m k = < / l a t e x i t > ✏ < l a t e x i t s h a 1 _ b a s e 6 4 = " a p r 4 O 2 l u L 3 I b + p K 1 I Z V Z a a j k / f w = " > A A A C A X i c b V D L S s N A F J 3 4 r P U V d S O 4 C R b B V U l E q s u i L g S 7 q G A f 0 I Q w m U 7 a o Z M H M z d i C X H j r 7 h x o Y h b / 8 K d f + O k z U J b D 1 w 4 n H M v 9 9 7 j x Z x J M M 1 v b W F x a X l l t b R W X t / Y 3 N r W d 3 b b M k o E o S 0 S 8 U h 0 P S w p Z y F t A Q N O u 7 G g O P A 4 7 X i j y 9 z v 3 F M h W R T e w T i m T o A H I f M Z w a A k V 9 + 3 A w x D g n n a y N z U B v o A 6 U 3 j K s t c v W J W z Q m M e W I V p I I K N F 3 9 y + 5 H J A l o C I R j K X u W G Y O T Y g G M c J q V 7 U T S G J M R H t C e o i E O q H T S y Q e Z c a S U v u F H Q l U I x k T 9 P Z H i Q M p x 4 K n O / F 4 5 6 + X i f 1 4 v A f / c S V k Y J 0 B D M l 3 k J 9 y A y M j j M P p M U A J 8 r A g m g q l b D T L E A h N Q o Z V V C N b s y / O k f V K 1 a t X a 7 W m l f l H E U U I H 6 B A d I w u d o T q 6 R k 3 U Q g Q 9 om f 0 i t 6 0 J + 1 F e 9 c + p q 0 L W j G z h / 5 A + / w B P y e X a w = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " 5N A Q a C / F 5 v Y a N 5 G 3 M m w f p c U B H 3 s = " > A A A C A X i c b V D L S g N B E J z 1 G e M r 6 k X w M h g E T 2 F X J H o M e v E Y M S 9 I Q p i d d J I h s 7 P L T K 8 Y l n j x V 7 x 4 U M S r f + H N v 3 E 2 y U E T C x q K q m 6 a K j + S w q D r f j t L y y u r a + u Z j e z m 1 v b O b m 5 v v 2 b C W H O o 8 l C G u u E z A 1 I o q K J A C Y 1 I A w t 8 C X V / e J 3 6 9 X v Q R o S q g q M I 2 g H r K 9 E T n K G V O r n D F s I D J n c R 0 w Z o R T N l R O q M O 7 m 8 W 3 A n o I v E m 5 E 8 m a H c y X 2 1 u i G P A 1 D I J T O m 6 b k R t h O m U X A J 4 2 w r N h A x P m R 9 a F q q W A C m n U w S j O m J V b q 0 F 2 o 7 C u l E / X 2 R s M C Y U e D b z Y D h w M x 7 q f i f 1 4 y x d 9 l O h I p i B M W n j 3 q x p B j S t A 7 a F R o 4 y p E l j G s b n V M + Y J p x t K V l b Q n e f O R F U j s r e M V C 8 f Y 8 X 7 q a 1 Z E h R + S Y n B K P X J A S u S F l U i W c P J J n 8k r e n C f n x X l 3 P q a r S 8 7 s 5 o D 8 g f P 5 A 1 K E l 3 g = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " V y 1 Our framework builds on VAE [34, 35] architecture, incorporating dedicate modules to handle nonstationarity. It enforces the conditions discussed in Sec. 3 as constraints. As shown in Fig. 2 , the framework consists of three primary components: (1) Sparse Transition, (2) Prior Network, andp d U x o 0 j 7 h Q 7 l V h Z 2 T r c m 7 J v g = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X J V E p L o s u n F Z w T 6 g C W E y n b R D Z y Z h Z i K U m I W / 4 s a F I m 7 9 D X f + j Z M 2 C 2 0 9 M H A 4 5 1 7 u m R M m j C r t O N 9 W Z W V 1 b X 2 j u l n b 2 t 7 Z 3 b P 3 D 7 o q T i U m H R y z W P Z D p A i j g n Q 0 1 Y z 0 E 0 k Q D x n p h Z O b w u 8 9 E K l o L O 7 1 N C E + R y N B I 4 q R N l J g H 3 m Z N 0 Y 6 8 z j S 4 z D K e J 4 H q Z c H d t 1 p O D P A Z e K W p A 5 K t A P 7 y x v G O O V E a M y Q U g P X S b S f I a k p Z i S v e a k i C c I T N C I D Q w X i R P n Z L H 8 O T 4 0 y h F E s z R M a z t T f G x n i S k 1 5 a C a L m G r R K 8 T / v E G q o y s / o y J J N R F 4 f i h K G d Q x L M q A Q y o J 1 m x q C M K S m q w Q j 5 F E W J v K a q Y E d / H L y 6 R 7 3 n C b j e b d R b 1 1 X d Z R B c f g B J w B F 1 y C F r g F b d A B G D y C Z / A K 3 q w n 6 8 V 6 t z 7 m o x W r 3 D k E f 2 B 9 / g A m r 5 b a < / l a t e x i t > { mu } < l a t e x i t s h a 1 _ b a s e 6 4 = " H u Q E m b f j y W E W Y z e Q l e 1 M s m 9 z C C U = " > A A A C N X i c b V D J S g N B F O y J W 4 x b 1 K O X x i B E 0 D A j E j 0 G v X j w E M E s k B l D T 6 c n a d K z 0 P 1 G i M P 8 l B f / w 5 M e P C j i 1 V + w s x z M U t B Q V N W j 3 y s 3 E l y B a b 4 b m a X l l d W 1 7 H p u Y 3 N r e y e / u 1 d X Y S w p q 9 F Q h L L p E s U E D 1 g N O A j W j C Q j v i t Y w + 1 f D / 3 G I 5 O K h 8 E 9 D C L m + K Q b c I 9 T A l p q 5 2 / t H o H E 9 g n 0 X C / x 0 / Q h O b X S 4 k i N 0 z a c 4 K n A U 7 p Y S 0 B P H b f z B b N k j o D n i T U h B T R B t Z 1 / t T s h j X 0 W A B V E q Z Z l R u A k R A K n g q U 5 O 1 Y s I r R P u q y l a U B 8 p p x k d H W K j 7 T S w V 4 o 9 Q s A j 9 T / E w n x l R r 4 r k 4 O N 1 W z 3 l B c 5 L V i 8 C 6 d h A d R D C y g 4 4 + 8 W G A I 8 b B C 3 O G S U R A D T Q i V X O + K a Y 9 I Q k E X n d M l W L M n z 5 P 6 W c k q l 8 p 3 5 4 X K 1 a S O L D p A h 6 i I L H S B K u g G V V E N U f S M 3 t A n + j J e j A / j 2 / g Z R z P G Z G Y f T c H 4 / Q M 9 8 a 2 G < / l a t e x i t > m 1 (û t , ẑt , ẑt 1 )(3) Encoder-Decoder.The transition module in our framework is designed to estimate transition functions { mu } U u=1 and a clustering function Ĉ as specified in Eq. ( 7). As highlighted in Sec. 3, the primary objective of this module is to model the transitions in the latent space and minimize the empirical transition complexity. To achieve this, we implemented U different transition networks for various m(û t , •) and added sparsity regularization to the transition functions via a sparsity loss. A gating function with a (hard)-Gumbel-Softmax function was used to generate ût , which was then employed to select the corresponding transition network mût . This network was further used to calculate the transition loss, which is explained in detail in Sec. 4.2.Prior Network The Prior Network module aims to effectively estimate the prior distribution p(ẑ t,i | ẑt-1 , ût ). This is achieved by evaluating p(ẑ t | ẑt-1 , ût ) = p ϵi m-1 i (û t , ẑt,i , ẑt-1 ) ∂ m-1 i ∂ ẑt,i , where m-1 i (û t , •)is the learned holistic inverse dynamics model. To ensure the conditional independence of the estimated latent variables, p(ẑ t | ẑt-1 ), we utilize an isomorphic noise distribution for ϵ and aggregate all estimated component densities to obtain the joint distribution p(ẑ t | ẑt-1 , ût ) as shown in Eq. ( 14). Given the lower-triangular nature of the Jacobian, its determinant can be computed as the product of its diagonal terms. Detailed derivations is provided in Appendix S3.1.log p (ẑt | ẑt-1, ût) = n i=1 log p(εi | ût) Conditional independence + n i=1 log ∂ m-1 i ∂ ẑt,iLower-triangular Jacobian (14) Encoder-Decoder The third component is an Encoder-Decoder module that utilizes reconstruction loss to enforce the invertibility of the learned mixing function ĝ. Specifically, the encoder fits the demixing function ĝ-1 and the decoder fits the mixing function ĝ. 
optimization: 
 The first training objective of CtrlNS is to fit the estimated transitions with minimum transition complexity according to Eq. ( 7):Lsparse ≜ EDL( mû t (ẑt-1), ẑt)Transition lossEQUATIONwhere L(•, •) is a regression loss function to fit the transition estimations, and the sparsity loss is approximated via L 2 norm of the parameter in the transition estimation functions.Then the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written as follows (complete derivation steps are in Appendix S3.2):EQUATIONWe use mean-squared error for the reconstruction likelihood loss L Recon . The KL divergence L KLD is estimated via a sampling approach since with a learned nonparametric transition prior, the distribution does not have an explicit form. Specifically, we obtain the log-likelihood of the posterior, evaluate the prior log p (ẑ t | ẑt-1 , ût ) in Eq. ( 14), and compute their mean difference in the dataset as the KL loss:L KLD = E ẑt∼q(ẑt|xt) log q(ẑ t |x t ) -log p (ẑ t | ẑt-1 , ût ).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The code to reproduce the experiment and the figures are provided as mentioned in Appendix S3.3. 
experiments: 
 We assessed the identifiability performance of CtrlNS on both synthetic and real-world datasets.For synthetic datasets, where we control the data generation process completely, we conducted a comprehensive evaluation. This evaluation covers the full spectrum of unknown nonstationary causal temporal representation learning, including metrics for both domain variables and the latent causal processes. In real-world scenarios, CtrlNS was employed in video action segmentation tasks. The evaluation metrics focus on the accuracy of action estimation for each video frame, which reflects the identifiability of domain variables.Evaluation Metrics For domain variables, we assessed the clustering accuracy (Acc) to estimate discrete domain variables u t . As the label order in clustering algorithms is not predetermined, we selected the order that yielded the highest accuracy score. For the latent causal processes, we computed the mean correlation coefficient (MCC) between the estimated latent variables ẑt and the ground truth z t . The MCC, a standard measure in the ICA literature for continuous variables, assesses the identifiability of the learned latent causal processes. We adjusted the reported MCC values in Table 1 by multiplying them by 100 to enhance the significance of the comparisons.Baselines We compared our method with identifiable nonlinear ICA methods: (1) BetaVAE [36] , which ignores both history and nonstationarity information. (2) i-VAE [13] and TCL [10] , which leverage nonstationarity to establish identifiability but assume independent factors. (3) SlowVAE [21] and PCL [11] , which exploit temporal constraints but assume independent sources and stationary processes. (4) TDRL [23] , which assumes nonstationary causal processes but with observed domain indices. ( 5) HMNLICA [15] , which considers the unobserved nonstationary part in the data generation process but does not allow any causally related time-delayed relations. ( 6) NCTRL [25] , which extends HMNLICA to an autoregressive setting to allow causally related time-delayed relations in the latent space but still assumes a Markov chain on the domain variables.We generate synthetic datasets that satisfy our identifiability conditions in Theorems 1 and 2, detailed procedures are in Appendix S2.1. The primary findings are presented in Table 1 . Note: the MCC metric is consistently available in all methods; however, the Acc metric for u t is only applicable to methods capable of estimating domain variables u t . In the first row of Table 1 , we evaluated a recent nonlinear temporal ICA method, TDRL, providing ground truth u t to establish an upper performance limit for the proposed framework. The high MCC (> 0.95) indicates the model's identifiability. Subsequently, the table lists six baseline methods that neglect the nonstationary domain variables, with none achieving a high MCC. The remaining approaches, including our proposed CtrlNS, are able to estimate the domain variables u t and recover the latent variables. In particular, HMNLICA exhibits instability during training, leading to considerable performance variability. This instability stems from HMNLICA's inability to allow time-delayed causal relationships among hidden variables z t , leading to model training failure when the actual domain variables deviate from the Markov assumption. In contrast, NCTRL, which extends TDRL under the same assumption, demonstrates enhanced stability and performance over HMNLICA by accommodating transitions in z t . However, since they use incorrect assumption on the nonstationary domain variables, the performance of those methods can be even worse than methods which do not include the domain information. Nevertheless, considering the significant nonstationarity and deviation from the Markov properties, those methods struggled to robustly estimate either the domain variables or the latent causal processes. Compared to all baselines, our proposed CtrlNS reliably recovers both u t (MCC > 0.95) and z t (Acc > 95%), and the MCC is on par with the upper performance bound when domain variables are given, justifying it effectivess. (3) In Phase 3, as u t becomes clearly identifiable, the MCC of z t progressively improves, ultimately achieving full identifiability. This three-phase process aligns perfectly with our theoretical predictions. According to Theorem 1, phases 1 and 2 should exhibit suboptimal z t estimations, while sparsity constraints can still guide training and improve the accuracy for domain variables u t . Once the accuracy of u t approaches high, Theorem 2 drives the improvement in MCC for z t estimations, leading to the final achievement of full identifiability of both latent causal processes for z t and domain variables u t .Experiment Setup Our method was tested on the video action segmentation task to estimate actions (domain variables u t ). Following [37, 28] , we use the same weakly supervised setting utilizing meta-information, such as action order. The evaluation included several metrics: Mean-over-Frames (MoF), the percentage of correctly predicted labels per frame; Intersection-over-Union (IoU), defined Datasets Our evaluation used two datasets: Hollywood Extended [38] , which includes 937 videos with 16 daily action categories, and CrossTask [39] , focusing on 14 of 18 primary tasks related to cooking [40] , comprising 2552 videos across 80 action categories.Model Design Our model is build on top of ATBA [28] method which uses multi-layer transformers as backbone networks. We add our sparse transition module with the sparsity loss function detailed in Sec. 4.2. Specifically, we integrated a temporally latent transition layer into ATBA's backbone, using a transformer layer across time axis for the Hollywood dataset and an LSTM for the CrossTask dataset. To encourage sparsity in the latent transitions, L 2 regularization is applied to the weights of the temporally latent transition layer. Result and Analysis The primary outcomes for real-world applications in action segmentation are summarized in Table 2 . Traditional methods based on hidden Markov models, such as HMM+RNN [41] and NN-Viterbi [43] , face challenges in these realworld scenarios. This observation corroborates our previous discussions on the limitations of earlier identifiability methods [15, 24, 25] , which depend on the Markov assumption for domain variables. Our approach significantly outperforms the baselines in both the Hollywood and CrossTask datasets across most metrics. Especially in the Hollywood dataset, our method outperforms the base ATBA model by quite a large margin. Notably, the Mean-over-Frames (MoF) metric aligns well with our identifiability results for domain variables u t . Our method demonstrates substantial superiority in this metric. For Intersectionover-Union (IoU) and Intersection-over-Detection (IoD), our results are comparable to those of the baseline methods in the CrossTask dataset and show its superiority in the Hollywood dataset. Furthermore, our proposed sparse transition module which aligns with human intuition and is easily integrated into existing methods like a plug-in module, thus further enhancing its impact in real-world scenarios.To make the illustration more straightforward, some example segmentation results from the Hollywood dataset are visualized in Figure 4 . By comparing the number of distinct actions and the segmentation boundaries between our method and the baseline, it is evident that our CtrlNS estimates the actions more accurately, demonstrating improved performance. Abalation Study Furthermore, we conducted an ablation study on the sparse transition module, as detailed in Table 3 . In this study, we test on a subset of Hollywood dataset for computational efficiency. For methods we compared, "-Complexity" refers to the configuration where we retain the latent transition layers but omit the sparse transition complexity regularization term from these layers, and "-Module" indicates the removal of the entire sparse transition module, effectively reverting the model to the baseline ATBA model. The comparative results in Table 3 demonstrate that both the dedicated design of the sparse transition module and the complexity regularization term enhance the performance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: The code and the data to reproduce the experiment and the figures are provided as mentioned in Appendix S3.3.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this study, we developed a comprehensive identifiability theory tailored for general sequential data influenced by nonstationary causal processes under unspecified distributional changes. We then introduced CtrlNS, a principled approach to recover both latent causal variables with their timedelayed causal relations, as well as determining the values of domain variables from observational data without relying on distributional or structural prior knowledge. Our experimental results demonstrate that CtrlNS can reliably estimate the domain indices and recover the latent causal process. And such module can be easily adapted to handle real-world scenarios such as action segmentation task.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experiment settings for both synthetic and real-world experiment are discussed in setup parts in Sec. 5. The Appendix. S3.4 also discussed the training details. Those information can also be easily obtained from the released code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitations: 
 As noted in Sec. 3.2, our main theorem relies on the condition that causal graphs among different domains must be distinct. Although our experiments indicate that this assumption is generally sufficient, there are scenarios in which it may not hold, meaning that the transition causal graphs are identical for two different domains, but the actual transition functions are different. We have addressed this partially through an extension to the mechanism variability assumption to higher-order cases (Corollary 1). However, dealing with situations where transition graphs remain the same across all higher orders remains a challenge. We acknowledge this as a limitation and suggest it as an area for future exploration. We also observed that the random initialization of the nonlinear ICA framework can influence the total number of epochs needed to achieve identifiability, as illustrated in Figure 3 . Also, for the computational efficiency, the TDRL framework we adopted involves a prior network that calculated each dimension in the latent space one by one, thus making the training efficiency suboptimal. Since this is not directly related to major claim which is our sparse transition design, we acknowledge this as a limitation and leave it for future work.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The synthetic experiments were supported by mean values and one standard derivation from the mean, with analysis on the stability of the training. For real-world setting, we report the error bars for our model. We run the experiments following existing setting on the data splitting or random seeds to calculate the mean and std. 
boarder impacts: 
 This work proposes a theoretical analysis and technical methods to learn the causal representation from time-series data, which facilitate the construction of more transparent and interpretable models to understand the causal effect in the real world. This could be beneficial in a variety of sectors, including healthcare, finance, and technology. In contrast, misinterpretations of causal relationships could also have significant negative implications in these fields, which must be carefully done to avoid unfair or biased predictions.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The codebase together with the datasets used in our paper are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Details about the code together with the synthetic dataset can be found in anonymized url provided in Appendix S3.3.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We divide the complete proof into two principal steps:1. Firstly, assuming access to the optimal mixing function estimation ĝ * , we demonstrate that under the conditions in our theorem, the estimated clustering result will align with the ground truth up to label swapping. This alignment is due to the transition complexity with optimal û * t and ĝ * being strictly lower than that with non-optimal ût but still optimal ĝ * . 2. Secondly, we generalize the results of the first step to cases where the mixing function estimation ĝ is suboptimal. We establish that for any given clustering assignment, whether optimal or not, a suboptimal mixing function estimation ĝ can not result in a lower transition complexity. Thus, the transition complexity in scenarios with non-optimal ĝ will always be at least as high as in the optimal case.From those two steps, we conclude that the global minimum transition complexity can only be attained when the estimation of domain variables ût is optimal, hence ensuring that the estimated clustering must match the ground truth up to label swapping. It is important to note that this condition alone does not guarantee the identifiability of the mixing function g. Because a setting with optimal û * t and a non-optimal ĝ may exhibit equivalent transition complexity to the optimal scenario, but it does not compromise our proof for the identifiability of domain variables u t . Further exploration of the mixing function's identifiability g is discussed in Theorem 2 in the subsequent section. We fist introduce a lemma for this case when we can access an optimal mixing function estimation ĝ * . Lemma S1 (Identifiability of C under optimal ĝ * ). In addition to the assumptions in Theorem 1, assume that we can also access an optimal estimation of g, denoted by ĝ * , in which the estimated ẑt is an invertible, component-wise transformation of a permuted version of z t . Then the estimated clustering Ĉ must match the ground truth up to label swapping.S1.1.1 Identifiability of C under optimal ĝ * < l a t e x i t s h a 1 _ b a s e 6 4 = " I C A 8 a c 8 C D N 7 7 R 4 V 7 F V X T l H f t Z m c = " > A A A C B 3 i c b V B N S 8 N A E J 3 4 W e t X 1 K M g i 0 X w V B K R 2 m P B i 8 c K 9 g O a U D b b T b t 0 s w m 7 G 6 G E 3 r z 4 V 7 x 4 U M S r f 8 G b / 8 Z N m 0 N t f T D w e G + G m X l B w p n S j v N j r a 1 v b G 5 t l 3 b K u 3 v 7 B 4 f 2 0 X F b x a k k t E V i H s t u g B X l T N C W Z p r T b i I p j g J O O 8 H 4 N v c 7 j 1 Q q F o s H P U m o H + G h Y C E j W B u p b 5 9 5 E d Y j g n n W n S J P s 4 g q t C D 1 7 Y p T d W Z A q 8 Q t S A U K N P v 2 t z e I S R p R o Q n H S v V c J 9 F + h q V m h N N p 2 U s V T T A Z 4 y H t G S q w W e h n s z + m 6 M I o A x T G 0 p T Q a K Y u T m Q 4 U m o S B a Y z P 1 E t e 7 n 4 n 9 d L d V j 3 M y a S V F N B 5 o v C l C M d o z w U N G C S E s 0 n h m A i m b k V k R G W m G g T X d m E 4 C 6 / v E r a V 1 W 3 V q 3 d X 1 c a 9 S K O E p z C O V y C C z f Q g D t o Q g s I P M E L v M G 7 9 W y 9 W h / W 5 7 x 1 z S p m T u A P r K 9 f Z K y Z o g = = < / l a t e x i t > X ⇥ Xk Y h U J 6 A a B Z f Y N N w I 7 M Q K a R g I b A e T u 7 n f f k K l e S Q f z D R G P 6 Q j y Y e c U W O l x m 2 / W H L L 7 g J k n X g Z K U G G e r / 4 1 R t E L A l R G i a o 1 l 3 P j Y 2 f U m U 4 E z g r 9 B K N M W U T O s K u p Z K G q P 1 0 c e i M X F h l Q I a R s i U N W a i / J 1 I a a j 0 N A 9 s Z U j P W q 9 5 c / M / r J m Z Y 9 V M u 4 8 S g Z M t F w 0 Q Q E 5 H 5 1 2 T A F T I j p p Z Q p r i 9 l b A x V Z Q Z m 0 3 B h u C t v r x O W l d l r 1 K u N K 5 L t W o W R x 7 O 4 B w u w Y M b q M E 9 1 K E J D B C e 4 Rk Y h U J 6 A a B Z f Y N N w I 7 M Q K a R g I b A e T u 7 n f f k K l e S Q f z D R G P 6 Q j y Y e c U W O l x m 2 / W H L L 7 g J k n X g Z K U G G e r / 4 1 R t E L A l R G i a o 1 l 3 P j Y 2 f U m U 4 E z g r 9 B K N M W U T O s K u p Z K G q P 1 0 c e i M X F h l Q I a R s i U N W a i / J 1 I a a j 0 N A 9 s Z U j P W q 9 5 c / M / r J m Z Y 9 V M u 4 8 S g Z M t F w 0 Q Q E 5 H 5 1 2 T A F T I j p p Z Q p r i 9 l b A x V Z Q Z m 0 3 B h u C t v r x O W l d l r 1 K u N K 5 L t W o W R x 7 O 4 B w u w Y M b q M E 9 1 K E J D B C e 4 RP L F J U s u v N C s h Q 6 Q r m z K f 9 z B h M = " > A A A B + n i c d V D L S g M x F M 3 4 r P U 1 1 a W b Y B F c D Z n a 1 u m u 0 I 3 L C v Y B n a F k 0 r Q N z T x I M k o Z 5 1 P c u F D E r V / i z r 8 x 0 1 Z Q 0 Q O B w z n 3 c k + O H 3 M m F U I f x t r 6 x u b W d m G n u L u 3 f 3 B o l o 6 6 M k o E o R 0 S 8 U j 0 f S w p Z y H t K K Y 4 7 c e C 4 s D n t O f P W r n f u 6 V C s i i 8 U f O Y e g G e h G z M C F Z a G p o l d 4 p V 6 g Z Y T Q n m a S v L h m Y Z W Y 0 G q t o N i K w a Q h W n r g m 6 q D i 1 G r Q t t E A Z r N A e m u / u K C J J Q E N F O J Z y Y K N Y e S k W i h F O s 6 K b S B p j M s M T O t A 0 x A G V X r q I n s E z r Y z g O B L 6 h Q o u 1 O 8 b K Q 6 k n A e + n s w z y t 9 e L v 7 l D R I 1 d r y U h X G i a E i W h 8 Y J h y q C e Q 9 w x A Q l i s 8 1 w U Q w n R W S K R a Y K N 1 W U Z f w 9 V P 4 P + l W L L t u 1 a + r 5 a a z q q M A T s A p O A c 2 u A R N c A X a o A M I u A M P 4 A k 8 G / f G o / F i v C 5 H 1 4 z V z j H 4 A e P t E z 1 d l K Q = < / l a t e x i t > ĈProof. In the first case we deal with optimal estimation ĝ * in which the estimated ẑt is an invertible, component-wise transformation of a permuted version of z t , but inaccurate estimated version of Ĉ, Consider the following example (Figure S1 ):With slight abuse of notation, we use C(A) to represent the domain assigned by C to all elements in A, and all elements in A have the same assignment. The same argument applies to B.Then, for an estimated Ĉ, if it incorrectly assigns two subsets of input A and B to the same û (Figure S1 orange circle), i.e.,EQUATION)Note that if the ground truth C gives a consistent assignment for A and B but estimated Ĉ gives diverse assignments, i.e.EQUATIONit is nothing but further splitting the ground truth assignment in a more fine-grained manner. This scenario does not break the boundaries of the ground truth assignments. Consider two cases in the estimation process:1. If the number of allowed regimes or domains exceeds that of the ground truth, such more fine-grained assignment is allowed. The ground truth can then be easily recovered by merging domains that share identical Jacobian supports.2. If the number of regimes or domains matches the ground truth, it can be shown that the inconsistent scenario outlined in Eq. ( 17) must occur.Given that these considerations do not directly affect our approach, they are omitted from further discussion for brevity. Then considering the case in Eq. 17, the estimated transition must cover the functions from both A and B, then the learned transition mk must have Jacobian J mk with support matrix M k = M i +M j which is the binary addition of M i and M j , such that for all indices in M i , M j if any of these two is 1, then the corresponding position in M k must be 1. That is because if that is not the case, for example, the (a, b)-th location for M i , M j , and M k are 1, 0, and 0. Then we can easily find an input region for the (a, b)-th location such that a small perturbation can lead to changes in m i but not in m j nor mk , which makes mk unable to fit all of the transitions in A ∪ B which cause contradiction. See the three matrices in Figure S2 for an illustrative example.M i =    0 0 1 0 1 1 0 0 0 0 0 1 0 0 1 0    M j =    1 0 1 0 1 1 0 0 0 0 0 1 0 0 1 0    M k =    1 0 1 0 1 1 0 0 0 0 0 1 0 0 1 0   By assumption iii, since all those support matrix differ at least one spot, which means the estimated version is not smaller than the ground truth.EQUATIONand the equality cannot hold true at the same time.Then from Assumption (i), the expected estimated transition complexity can be expressed as:EQUATIONSimilarly for ground truth one:EQUATIONLet us focus on the integral of the region A ∪ B, the subset of X × X mentioned above. If for some area p D (x t-1 , x t ) = 0, then the clustering under this area is ill defined since there is no support from data. Hence we only need to deal with supported area. For area that p D (x t-1 , x t ) > 0 and from Eq. ( 19) the equality cannot hold true at the same time, then the estimated version of the integral is strictly larger than the ground-truth version for any inconsistent clustering as indicated in Eq. ( 17).For the rest of regions in X × X , any incorrect cluster assignment will further increase the M with same reason as discussed above, then the estimated complexity is strictly larger than the ground truth complexity:EQUATIONBut assumption (ii) requires that the estimated complexity be less than or equal to the ground truth. Contradiction! Hence, the estimated Ĉ must match the ground truth up to label swapping.Now we can leverage the conclusion in Lemma S1 to show the identifiability of domain variables under arbitrary mixing function estimation. Theorem S1 (Identifiability of Domain Variables). Suppose that the data D are generated from the nonstationary data generation process as described in Eqs. (1) and (2). Suppose the transitions are weakly diverse lossy (Def. 6) and the following assumptions hold:i. (Mechanism Separability) There exists a ground truth mapping C : X × X → U determined the real domain indices, i.e., u t = C(x t-1 , x t ).ii. (Mechanism Sparsity) The estimated transition complexity on dataset D is less than or equal to ground truth transition complexity, i.e.,E D | M û| ≤ E D |M u |.iii. (Mechanism Variability) Mechanisms are sufficiently different. For all u ̸ = u ′ , M u ̸ = M u ′ i.e. there exists index (i, j) such that[M u ] i,j ̸ = [M u ′ ] i,j .Then the domain variables u t is identifiable up to label swapping (Def. 2).Proof. To demonstrate the complete identifiability of C, independent of the estimation quality of g, we must show that for any arbitrary estimation Ĉ ̸ = σ(C), the induced Mû for inaccurate estimation ĝ has a transition complexity at least as high as in the optimal ĝ * case. If this holds, from Lemma S1, we can conclude that the transition complexity of optimal Ĉ * = σ(C) and optimal ĝ * is strictly smaller than any non-optimal Ĉ and any ĝ.Suppose the estimated decoder and corresponding latent variables are ĝ and ẑt , respectively, then the following relation holds:EQUATION)Since ĝ is invertible, by composing ĝ-1 on both sides, we obtain:EQUATIONLetEQUATIONwe then have:EQUATIONWe aim to demonstrate that under this transformation, if h is not a permutation and component-wise transformation, the introduced transition complexity among estimated ẑ will not be smaller than the optimal ĝ * . Proposition S1. Suppose | M| < | M * |, then for any permutation σ mapping the indices of the dimensions from M to M * , there must exist an index pair (i, j) such that M i,j = 0 and M * σ(i),σ(j) = 1.Proof. An intuitive explanation for this proposition involves the construction of a directed graphG M * = (V M * , E M * ), where V M * = {1, 2, . . . , n} and E M * = {(i, j) | M * i,j = 1}. A similar construction can be made for G M . It is straightforward that | M * | = |E M * |, which represents the number of edges. Consequently, | M| < | M * | implies that G M * has more edges than G M .Since there is no pre-defined ordering information for the nodes in these two graphs, if we wish to compare their edges, we need to first establish a mapping. However, if |E M | < |E M * |, no matter how the mapping σ is constructed, there must be an index pair (i, j) such that (i, j) / ∈ E M but (σ(i), σ(j)) ∈ E M * . Otherwise, if such an index pair does not exist, the total number of edges in G M would necessarily be greater than or equal to that in G M * , contradicting the premise that| M| < | M * |.Lemma S2 (Non-decreasing Complexity). Suppose transitions are weakly diverse lossy as defined in Def. 6 and an invertible transformation h maps the optimal estimation ẑ * t to the estimated ẑt , and it is neither a permutation nor a component-wise transformation. Then, the transition complexity on the estimated ẑt is not lower than that on the optimal ẑ * t , i.e., | M| ≥ | M * |.Proof. The entire proof is based on contradiction. In Figure S3 , we provide an illustrative example. Note that the mapping from ground truth z t to optimal estimation ẑ * t is a permutation and elementwise transformation, it does not include mixing, and hence e i exists if and only if ê * i exists. Therefore, | M * | = |M|. The core of the proof requires us to demonstrate that | M| < |M| cannot be true.Then, according to this lemma, the transition complexity | M û| of the learned mû should be greater than or equal to | M * û|, which is the complexity when using an accurate estimation of ĝ * . This relationship can be expressed as follows:| M û| ≥ | M * û|.By Lemma S1, the expected complexity of the estimated model E D | M * û| is strictly larger than that of the ground truth E D |M u |. This implies the following inequality:EQUATIONHowever, Assumption (ii) requires that the estimated complexity must be less than or equal to the ground-truth complexity, leading to a contradiction. This contradiction implies that the estimated Ĉ must match the ground truth up to label swapping. Consequently, this supports the conclusion of Theorem 1.Corollary S1 (Identifiability under Function Variability). Suppose the data D is generated from the nonstationary data generation process described in (1) and (2). Assume the transitions are weakly diverse lossy (Def. 6), and the mechanism separability assumption i along with the following assumptions hold:v. (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K ∈ N such that for all u ̸ = u ′ , there existsk ≤ K, M k u ̸ = M k u ′ i.e. there exists index (i, j) such that M k u i,j ̸ = M k u ′ i,j .vi. (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset D is no more than ground truth transition complexity,EQUATION)Then the domain variables u t are identifiable up to label swapping (Def. 2).Proof. With a strategy similar to the proof of Theorem 1, we aim to demonstrate that using an incorrect cluster assignment Ĉ will result in K k=1 |M k û| being higher than the ground truth, thereby still enforcing the correct u t .Differing slightly from the approach in Theorem 1, in this setting, we will first demonstrate that under any arbitrary Ĉ assignment, the estimated complexity is no lower than the complexity in the ground truth, i.e.,K k=1 | M k | ≥ K k=1 |M k |.First, we address the scenario where two different domains have the same transition graph but with different functions, as otherwise, the previous lemma S1 still applies. In cases where the same transition causal graph exists but the functions differ, assumption v indicates that there exists an integer k such that M k u ̸ = M k u ′ , meaning the ground truth support matrices are different. However, due to incorrect clustering, the learned transition must cover both cases. To substantiate this claim, we need to first introduce an extension of the non-decreasing complexity lemma.Lemma S3 (Non-decreasing Complexity under Mechanism Function Variability). Suppose there exists an invertible transformation h which maps the ground truth z t to the estimated ẑt , and it is neither a permutation nor a component-wise transformation. Then, the transition complexity on the estimated ẑt is not lower than that on the ground truth z t , i.e.,K k=1 | M k | ≥ K k=1 |M k |.Theorem S2 (Identifiability of the Latent Causal Processes). Suppose that the data D is generated from the nonstationary data generation process (1), (2) , which satisfies the conditions in Theorem 1 and Lemma 1, then the domain variables u t are identifiable up to label swapping (Def. 2) and latent variables z t are identifiable up to permutation and a component-wise transformation (Def. 3).Proof. From Theorem 1, the domain variables u t are identifiable up to label swapping, and then use the estimated domain variables in Lemma 1, the latent causal processes are also identifiable, that is, z t are identifiable up to permutation and a component-wise transformation.Note that we assume that there exists a ground truth mapping C : X × X → U, gives a domain index based on x t-1 , x t . The existence of such mapping means that the human can tell what the domain is based on two consecutive observations. If two observations are not sufficient, then it can be modified to have more observation steps as input, for example x ≤t or even full sequence x [1:T ] . If the input has future observation, which means that x >t is included, then this is only valid for sequence understanding tasks in which the entire sequence will be visible to the model when analyzing the time step t. For prediction tasks or generation tasks, further assumptions on C such as the input only contains x <t should be made, which will be another story. Those variants are based on specific application scenarios and not directly affect our theory, for brevity, let us assume the two-step case.This is a rather intuitive assumption in which we introduce some form of sparsity in the transitions, and our task is to ensure that the estimated transition maintains this sparsity pattern. This requirement is enforced by asserting an equal or lower transition complexity as defined in Assumption ii. Similar approaches, grounded in the same intuition, are also explored in the reinforcement learning setting, as discussed in works by Lachapelle et al. [18] and Hwang et al. [45] . The former emphasizes the identifiability result of the independent components, which necessitates additional assumptions. In contrast, the latter focuses on the RL scenario, requiring the direct observation of the latent variables involved in the dynamics, which leaves significant challenges in real-world sequenceunderstanding tasks, where the states are latent. Some studies have also explored the application of sparsity constraints on the estimated latent representations [46] . And it is also extensively discussed in the nonlinear ICA literature [33, 19] , in which such a sparsity constraint was added to the mixing function.The assumption of mechanism variability requires that causal dynamics differ between domains, which requires at least one discrete edge variation within the causal transition graphs. This assumption is typically considered reasonable in practical contexts; humans identify distinctions between domains only when the differences are substantial, which often involves the introduction of a new mechanism or the elimination of an existing one. Specifically, this assumption requires a minimal alteration, a single edge change in the causal graph, to be considered satisfied. Consequently, as long as there are significant differences in the causal dynamics among domains, this criterion is fulfilled.In this section, we will further discuss the mechanism function variability introduced in Corollary 1. One might question the necessity of this assumption. To illustrate this issue, we claim that if we only assume that the mechanism functions differ across domains but without this extended version of the variability assumption, i.e., for u ̸ = u ′ , m u ̸ = m u ′ , then under this proposed framework, the domain variables u t are generally unidentifiable.In Figure S4 , we present a simple example of the space of z t+1 given a fixed z t . For the sake of brevity, assume that there are two domains. By the mechanism separability assumption i, the space Z of z t+1 is divided into two distinct parts, each corresponding to one domain. In this illustration, C denotes the partition created by the ground truth transition function:EQUATION)< l a t e x i t s h a 1 _ b a s e 6 4 = " t F Q 0 A < l a t e x i t s h a 1 _ b a s e 6 4 = " x W 4 O h l C h b a l 2 R z H y Q M S N u h g T X x A = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H a N Q Y 5 E L x 4 h k U c C G z I 7 N D A y O 7 u Z m T U h G 7 7 A i w e N 8 e o n e f N v H G A P C l b S S a W q O 9 1 d Q S y 4 N q 7 7 7 e Q 2 N r e 2 d / K 7 h b 3 9 g 8 O j 4 v F J S 0 e J Y t h k k Y h U J 6 A a B Z f Y N N w I 7 M Q K a R g I b A e T u 7 n f f k K l e S Q f z D R G P 6 Q j y Y e c U W O l x m 2 / W H L L 7 g J k n X g Z K U G G e r / 4 1 R t E L A l R G i a o 1 l 3 P j Y 2 f U m U 4 E z g r 9 B K N M W U T O s K u p Z K G q P 1 0 c e i M X F h l Q I a R s i U N W a i / J 1 I a a j 0 N A 9 s Z U j P W q 9 5 c / M / r J m Z Y 9 V M u 4 8 S g Z M t F w 0 Q Q E 5 H 5 1 2 T A F T I j p p Z Q p r i 9 l b A x V Z Q Z m 0 3 B h u C t v r x O W l d l r 1 K u N K 5 L t W o W R x 7 O 4 B w u w Y M b q M E 9 1 K E J D B C e 4 R X e n E f n x X l 3 P p a t O S e b O Y U / c D 5 / A J T x j M c = < / l a t e x i t > B < l a t e x i t s h a 1 _ b a s e 6 4 = " m y x o H d S z r r X s F q F L z H V h e y W r M 7X f 3 i D L 8 / v 2 t A l u c q 7 y o D s Y o = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H a N Q Y 4 Y L x 4 h k U c C G z I 7 N D A y O 7 u Z m T U h G 7 7 A i w e N 8 e o n e f N v H G A P C l b S S a W q O 9 1 d Q S y 4 N q 7 7 7 e Q 2 N r e 2 d / K 7 h b 3 9 g 8 O j 4 v F J S 0 e J Y t h k k Y h U J 6 A a B Z f Y N N w I 7 M Q K a R g I b A e T u 7 n f f k K l e S Q f z D R G P 6 Q j y Y e c U W O l x m 2 / W H L L 7 g J k n X g Z K U G G e r / 4 1 R t E L A l R G i a o 1 l 3 P j Y 2 f U m U 4 E z g r 9 B K N M W U T O s K u p Z K G q P 1 0 c e i M X F h l Q I a R s i U N W a i / J 1 I a a j 0 N A 9 s Z U j P W q 9 5 c / M / r J m Z Y 9 V M u 4 8 S g Z M t F w 0 Q Q E 5 H 5 1 2 T A F T I j p p Z Q p r i 9 l b A x V Z Q Z m 0 3 B h u C t v r x O W l d l r 1 K u N K 5 L t W o W R x 7 O 4 B w u w Y M b q M E 9 1 K E J D B C e 4 RI = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H a N Q Y 4 k X D x C I o 8 E N m R 2 a G B k d n Y z M 2 t C N n y B F w 8 a 4 9 V P 8 u b f O M A e F K y k k 0 p V d 7 q 7 g l h w b V z 3 2 8 l t b e / s 7 u X 3 C w e H R 8 c n x d O z t o 4 S x b D F I h G p b k A 1 C i 6 x Z b g R 2 I 0 V 0 j A Q 2 A m m 9 Y X f e U K l e S Q f z C x G P 6 R j y U e c U W O l Z n 1 Q L L l l d w m y S b y M l C B D Y 1 D 8 6 g 8 j l o Q o D R N U 6 5 7 n x s Z P q T K c C Z w X + o n G m L I p H W P P U k l D 1 H 6 6 P H R O r q w y J K N I 2 Z K G L N X f E y k N t Z 6 F g e 0 M q Z n o d W 8 h / u f 1 E j O q + i m X c W J Q s t W i U S K I i c j i a z L k C p k R M 0 s o U 9 z e S t i E K s q M z a Z g Q / D W X 9 4 k 7 Z u y V y l X m r e l W j W L I w 8 X c A n X 4 M E d 1 O A e G t A C B g j P 8 A p v z q P z 4 r w 7 H 6 v W n J P N n M M f O J 8 / l n W M y A = = < / l a t e x i t >C < l a t e x i t s h a 1 _ b a s e 6 4 = " o e i f 9 j 4 p n u A w R Y r + j H r / R S 8D d m k = " > A A A B 6 H i c b V D L S g N B E O y N r x h f U Y 9 e B o P g K e y K x B w D e v C Y g H l A s o T Z S S c Z M z u 7 z M w K Y c k X e P G g i F c / y Z t / 4 y T Z g y Y W N B R V 3 X R 3 B b H g 2 r j u t 5 P b 2 N z a 3 s n v F v b 2 D w 6 P i s c n L R 0 l i m G T R S J S n Y B q F F x i 0 3 A j s B M r p G E g s B 1 M b u d + + w m V 5 p F 8 M N M Y / Z C O J B 9 y R o 2 V G n f 9 Y s k t u w u Q d e J l p A Q Z 6 v 3 i V 2 8 Q s S R E a Z i g W n c 9 N z Z + S p X h T O C s 0 E s 0 x p R N 6 A i 7 l k o a o v b T x a E z c m G V A R l G y p Y 0 Z K H + n k h p q P U 0 D G x n S M 1 Y r 3 p z 8 T + v m 5 h h 1 U + 5 j B O D k i 0 X D R N B T E T m X 5 M B V 8 i M m F p C m e L 2 V s L G V F F m b D Y F G 4 K 3 + v I 6 a V 2 V v U q 5 0 r g u 1 a p Z H H k 4 g 3 O 4 B A 9 u o A b 3 U I c m M E B 4 h l d 4 cx 6 d F + f d + V i 2 5 p x s 5 h T + w P n 8 A Z f 5 j M k = < / l a t e x i t > D < l a t e x i t s h a 1 _ b a s e 6 4 = " K F Then the question arises: when the domain assignment is incorrect, that is, Ĉ ̸ = C, can we still get the same observational distributions, or equivalently, can we obtain the same distribution for z t+1 ?P L F J U s u v N C s h Q 6 Q r m z K f 9 z B h M = " > A A A B + n i c d V D L S g M x F M 3 4 r P U 1 1 a W b Y B F c D Z n a 1 u m u 0 I 3 L C v Y B n a F k 0 r Q N z T x I M k o Z 5 1 P c u F D E r V / i z r 8 x 0 1 Z Q 0 Q O B w z n 3 c k + O H 3 M m F U I f x t r 6 x u b W d m G n u L u 3 f 3 B o l o 6 6 M k o E o R 0 S 8 U j 0 f S w p Z y H t K K Y 4 7 c e C 4 s D n t O f P W r n f u 6 V C s i i 8 U f O Y e g G e h G z M C F Z a G p o l d 4 p V 6 g Z Y T Q n m a S v L h m Y Z W Y 0 G q t o N i K w a Q h W n r g m 6 q D i 1 G r Q t t E A Z r N A e m u / u K C J J Q E N F O J Z y Y K N Y e S k W i h F O s 6 K b S B p j M s M T O t A 0 x A G V X r q I n s E z r Y z g O B L 6 h Q o u 1 O 8 b K Q 6 k n A e + n s w z y t 9 e L v 7 l D R I 1 d r y U h X G i a E i W h 8 Y J h y q C e Q 9 w x A Q l i s 8 1 w U Q w n R W S K R a Y K N 1 W U Z f w 9 V P 4 P + l W L L t u 1 a + r 5 a a z q q M A T s A p O A c 2 u A R N c A X a o A M I u A M P 4 A k 8 G / f G o / F i v C 5 H 1 4 z V z j H 4 A e P t E z 1 d l K Q = < / l a t e x i t > Ĉ < l a t e x i t s h a 1 _ b a s e 6 4 = " A T i i s W y 1 p / + p k 2 J 4 I a d f h E 6 o k 1 4 = " > A A A B 8 n i c d V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 G m b q 9 L U r d O O y g r W F 6 V A y a a Y N z S R D k h H K 0 M 9 w 4 0 I R t 3 6 N O / / G T F t B R Q 8 E D u f c S 8 4 9 Y c K o 0 o 7 z Y R U 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v H J n R K p x K S H B R N y E C J F G O W k p 6 l m Z J B I g u K Q k X 4 4 6 + R + / 5 5 I R Q W / 1 f O E B D G a c B p R j L S R / G G M 9 B Q j l n U W o 3 L F s Z 1 a y 2 t U o W P X H L f l X R n S a j W 9 e g 2 6 t r N E B a z R H Z X f h 2 O B 0 5 h w j R l S y n e d R A c Z k p p i R h a l Y a p I g v A M T Y h v K E c x U U G 2 j L y A F 0 Y Z w 0 h I 8 7 i G S / X 7 R o Z i p e Z x a C b z i O q 3 l 4 t / e X 6 q o 2 a Q U Z 6 k m n C 8 + i h K G d Q C 5 v f D M Z U E a z Y 3 B G F J T V a I p 0 g i r E 1 L J V P C 1 6 X w f 3 J X t d 2 6 X b / x K u 3 m u o 4 i O A P n 4 B K 4 o A H a 4 B p 0 Q Q 9 g I M A D e A L P l r Y e r R f r d T V a s N Y 7 p + A H r L d P 9 b G R t A = = < / l a t e x i t > C < l a t e x i t s h a 1 _ b a s e 6 4 = " x 2 x Y C a O o U d g z z w K J y 0 8 o O N W R A 4 A = " > A A A B 8 n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j M i t c u C G 5 c V 7 A O n Q 8 m k m T Y 0 k w z J H a E M / Q w 3 L h R x 6 9 e 4 8 2 / M t L P Q 1 g O B w z n 3 k n N P m A h u w H W / n d L G 5 t b 2 T n m 3 s r d / c H h U P T 7 p G p V q y j p U C a X 7 I T F M c M k 6 w E G w f q I Z i U P B e u H 0 N v d 7 T 0 w b r u Q D z B I W x G Q s e c Q p A S v 5 g 5 j A h B K R P c 6 H 1 Z p b d x f A 6 8 Q r S A 0 V a A + r X 4 O R o m n M J F B B j P E 9 N 4 E g I x o 4 F W x e G a S G J Y R O y Z j 5 l k o S M x N k i 8 h z f G G V E Y 6 U t k 8 C X q i / N z I S G z O L Q z u Z R z S r X i 7 + 5 / k p R M 0 g 4 z J J g U m 6 / C h K B Q a F 8 / v x i G t G Q c w s I V R z m x X T C d G E g m 2 p Y k v w V k 9 e J 9 2 r u t e o N + 6 v a 6 1 m U U c Z n a F z d I k 8 d I N a 6 A 6 1 U Q d R p N A z e k V v D j g v z r v z s R w t O c X O K f o D 5 / M H l t u R c Q = = <The answer is yes. For the ground truth transition,m 1 (z t , ϵ) ∈ A ∪ C and m 2 (z t , ϵ) ∈ B ∪ D.In the case of an incorrect partition Ĉ, it is sufficient to have m1 (z t , ϵ) ∈ A ∪ B and m2 (z t , ϵ) ∈ C ∪ D. Ensuring that the conditional distribution p(z t+1 | z t ) is matched everywhere, we can create two different partitions on domains, yet still obtain exactly the same observations. That makes the domain variables u t unidentifiable in the general case.How does the previous mechanism variability assumption work? In the assumption of mechanism variability (Assumption iii), the support matrices of the Jacobian of transitions across different domains differ. Consider a scenario where the ground truth partition is C, denoted by A, C | B, D. If an incorrect estimation occurs, where our estimated partition is Ĉ, represented as A, B | C, D, then the estimated transition in domain one should cover the transitions in both A and B, and similarly for the second domain. This leads to an increase in complexity within the estimated Jacobian support matrix, as discussed in the previous sections. Consequently, this complexity forces the sets B and C to be empty, resulting in Ĉ converging to C.How about mechanism function variability? Roughly speaking, and as demonstrated in our experiments, the mechanism variability assumption previously discussed is already sufficient to identify domain changes in both synthetic and real-world settings. This sufficiency arises because the assumption only requires a single differing spot, even though some transition functions behind some edges may persist across different domains. As long as there is one edge spot that can separate the two domains, this condition is met. In the relatively rare case where all edges in the causal dynamic transition graphs are identical across two different domains and only the underlying functions differ, we can still demonstrate identifiability in this scenario by examining differences in the support of the higher-order partial derivative matrices.The weakly diverse lossy transition assumption requires that each variable in the latent space can potentially influence a set of subsequent latent variables, and such transformations are typically non-invertible. This implies that given the value of z t+1 , it is generally challenging to precisely recover the previous z t ; equivalently, this mapping is not injective. Although this assumption requires some explanation, it is actually considered mild in practice. Often in real-world scenarios, different current states may lead to identical future states, indicating a loss of information. The "weakly diverse" of this assumption suggests that the way information is lost varies between different dimensions, but there is some common part among them, hence the term "weakly diverse". In the visualization example shown in Figure 4 , we can clearly see this pattern, in which the scene is relatively simple and it is very likely that in two different frames, the configuration of the scene or the value of the latent variables are the same but their previous states are completely different.The synthetic dataset is constructed in accordance with the conditions outlined in Theorems 1 and 2.Transition and mixing functions are synthesized using multilayer perceptrons (MLPs) initialized with random weights. The mixing functions incorporate the LeakyReLU activation function to ensure invertibility. The dataset features five distinct values for the domain variables, with both the hidden variables z t and the observed variables x t set to eight dimensions. A total of 1,000 sequences of domain variables were generated. These sequences exhibit high nonstationarity across domains, which cannot be captured with a single Markov chain. This was achieved by initially generating two distinct Markov chains to generate two sequences of domain indices. Subsequently, these sequences were concatenated, along with another sequence sampled from a discrete uniform distribution over the set {1, 2, 3, 4, 5}, representing the domain indices. For each sequence of domain variables, we sampled a batch size of 32 sequences of hidden variables z t beginning from a randomly initialized initial state z 0 . These sequences were generated using the randomly initialized multilayer perceptron (MLP) to model the transitions. Observations x t were subsequently generated from z t using the mixing function as specified in Eq. 1. Both the transition functions in the hidden space and the mixing functions were shared across the entire dataset. A summary of the statistics for this synthetic dataset is provided in Table S1 . For detailed implementation of this data generation process, please refer to our accompanying code in Sec. S3.3.Hollywood Extended The Hollywood [38] dataset contains 937 video clips with a total of 787,720 frames containing sequences of 16 different daily actions such as walking or sitting from 69 Hollywood movies. On average, each video comprises 5.9 segments, and 60.9% of the frames are background.CrossTask The CrossTask [39] dataset features videos from 18 primary tasks. According to [40] , we use the selected 14 cooking-related tasks, including 2552 videos with 80 action categories. On average, each video in this subset has 14.4 segments, with 74.8% of the frames classified as background.MCC, a standard metric in the ICA literature, is utilized to evaluate the recovery of latent factors. This method initially computes the absolute values of the correlation coefficients between each ground truth factor and every estimated latent variable. Depending on the presence of componentwise invertible nonlinearities in the recovered factors, either Pearson's correlation coefficients or Spearman's rank correlation coefficients are employed. The optimal permutation of the factors is determined by solving a linear sum assignment problem on the resultant correlation matrix, which is executed in polynomial time.Let us start with an illustrative example of stationary latent causal processes consisting of two timedelayed latent variables, i.e., z t = [z 1,t , z 2,t ], i.e., z i,t = m i (z t-1 , ϵ i,t ) with mutually independent noises, where we omit the u t since it is just an index to select the transition function m i . Let us write this latent process as a transformation map m (note that we overload the notation m for transition functions and for the transformation map):EQUATIONBy applying the change of variables formula to the map m, we can evaluate the joint distribution of the latent variables p(z 1,t-1 , z 2,t-1 , z 1,t , z 2,t ) as:EQUATIONwhere J m is the Jacobian matrix of the map m, which is naturally a low-triangular matrix:J m =     1 0 0 0 0 1 0 0 ∂z1,t ∂z1,t-1 ∂z1,t ∂z2,t-1 ∂z1,t ∂ϵ1,t 0 ∂z2,t ∂z1,t-1 ∂z2,t ∂z2,t-1 0 ∂z2,t ∂ϵ2,t     .Given that this Jacobian is triangular, we can efficiently compute its determinant as i ∂zi,t ∂ϵi,t . Furthermore, because the noise terms are mutually independent, and hence ϵ i,t ⊥ ϵ j,t for j ̸ = i and ϵ t ⊥ z t-1 , we can write the RHS of Eq. 31 as:EQUATIONFinally, by canceling out the marginals of the lagged latent variables p(z 1,t-1 , z 2,t-1 ) on both sides, we can evaluate the transition prior likelihood as:EQUATIONNow we generalize this example and derive the prior likelihood below.Let { m-1 i } i=1,2,3... be a set of learned inverse transition functions that take the estimated latent causal variables, and output the noise terms, i.e., εi,t = m-1 i (u t , ẑi,t , ẑt-1 ). Design transformation A → B with low-triangular Jacobian as follows:EQUATIONSimilar to Eq. 33, we can obtain the joint distribution of the estimated dynamics subspace as:log p(A) = log p (ẑt-1) + n j=1log p(εi,t)EQUATIONEQUATIONThen the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written as:EQUATIONS3. For synthetic experiments, the models were implemented in PyTorch 2.2.2. We trained the VAE network using the AdamW optimizer with a learning rate of 5 × 10 -4 and a mini-batch size of 64. Each experiment was conducted using three different random seeds and we reported the mean performance along with the standard deviation averaged across these seeds. The coefficient for the L2 penalty term was set to 1 × 10 -4 , which yielded satisfactory performance in our experiments.the most accurate way to enforce sparsity is through the L0 norm. However, since calculating the gradient for it is challenging, Lp norms are commonly used as approximations. We also tested alternative settings such as L1 penalty or larger coefficients for L2 norm, and we found that the setting we used in this paper (L2 with coefficient 1 × 10 -4 ) provided the best stability and performance.All other hyperparameters of the baseline methods follow their default values from their original implementation. For real-world experiments, we follow the same hyperparameter setting from the baseline ATBA method. In the Hollywood dataset, we used the default 10-fold dataset split setting and calculated the mean and standard derivation from those 10 runs. For the CrossTask dataset, we calculate the mean and standard derivation using five different random seeds.In our experimental setup, using reconstruction loss already provides strong identifiability results (MCC>0.95), and it is widely used in the identifiability literature. During our experiments, we found that flow-based methods are usually less efficient and typically take longer to converge. Since our main contribution is to address the challenge of unknown domain variables, this choice is orthogonal to our theoretical contribution. Therefore, we followed the existing work for the design of the estimation for the mixing function. As also mentioned in [47] , the reconstruction loss-based framework can definitely be extended to flow-based methods, especially in environments where invertibility is a critical issue and computation is not a top priority in the estimation process.As introduced in 4, we utilize U distinct networks to capture the different transitions. One may question whether this choice introduces redundant parameters and suggest employing parameter-sharing networks, as in [22, 45] . However, our decision to use separate transition networks is based on the assumptions regarding the complexity of the transition functions, which we regularize through the sparsity of the Jacobian matrix. Implementing parameter-sharing techniques would significantly complicate the optimization process, as updating parameters for one domain would immediately alter the Jacobian matrix for another domain.Despite using individual transition networks, they remain lightweight compared to the entire framework. Even in synthetic scenarios where the framework is relatively small, each transition network accounts for only approximately 2.3% of the total parameters. This proportion becomes even smaller in real-world applications with larger encoder-decoder frameworks.Various studies have focused on uncovering causally related latent variables. For example, [48] [49] [50] use vanishing Tetrad conditions [51] or rank constraints to detect latent variables in linear-Gaussian models, whereas [52] [53] [54] [55] rely on non-Gaussianity in their analyses of linear, non-Gaussian models. Additionally, some methods seek to identify structures beyond latent variables, leading to hierarchical structures. Certain hierarchical model-based approaches assume tree-like configurations, as seen in [56] [57] [58] [59] , while other methods consider a more general hierarchical structure [55, 50] . Nonetheless, these approaches are restricted to linear frameworks and encounter increasing difficulties with complex datasets, such as videos.In the context of sequence or time series data, recent advances in nonlinear Independent Component Analysis (ICA) have leveraged temporal structures and nonstationarities to achieve identifiability. Time-contrastive learning (TCL) [10] exploits variability in variance across data segments under the assumption of independent sources. Permutation-based contrastive learning (PCL) [11] discriminates between true and permuted sources using contrastive loss, achieving identifiability under the uniformly dependent assumption. The i-VAE [13] uses Variational Autoencoders to approximate the joint distribution over observed and nonstationary regimes. Additionally, (i)-CITRIS [60, 61] utilizes intervention target information to identify latent causal factors. Other approaches such as LEAP [22] and TDRL [23] leverage nonstationarities from noise and transitions to establish identifiability. GCIM [62] builds on the theoretical insights from LEAP by implementing a clustering algorithm to address the challenge of unobserved domain indices. However, it does not achieve identifiability of the domain variables. CaRiNG [63] extended TDRL to handle non-invertible generation processes by assuming sequence-wise recoverability of the latent variables from observations.All the aforementioned methods either assume stationary fixed temporal causal relations or that the domain variables controlling the nonstationary transitions are observed. To address unknown or unobserved domain variables, HMNLICA [15] integrates nonlinear ICA with a hidden Markov model to automatically model nonstationarity. However, this method does not account for the autoregressive latent transitions between latent variables over time. IDEA [26] combines HMNLICA and TDRL by categorizing the latent factors into domainvariant and domain-invariant groups. For the variant variables, IDEA adopts the same Markov chain model as HMNLICA, while for the invariant variables, it reduces the model to a stationary case handled by TDRL. Both iMSM [24] and NCTRL [25] extend this Markov structure approach by incorporating transitions in the latent space but continue to assume that the domain variables follow a Markov chain.Moreover, causal temporal representation learning has been explored in the reinforcement learning literature [64, 45] . These works primarily focus on the relationship between states and actions, typically relying on direct state observations. In contrast, our setting involves recovering meaningful latent causal variables from observational data.Weakly-supervised action segmentation techniques focus on dividing a video into distinct action segments using training videos annotated solely by transcripts [38, 65, 43, 66, 42, 67, 41, [68] [69] [70] [71] [72] [73] [74] . Although these methods have varying optimization objectives, many employ pseudo-segmentation for training by aligning video sequences with transcripts through techniques like Connectionist Temporal Classification (CTC) [65] ,Viterbi [41, 43, 42, 70, 71, 73] , or Dynamic Time Warping (DTW) [67, 69] . For instance, [65] extends CTC to consider visual similarities between frames while evaluating valid alignments between videos and transcripts. Drawing inspiration from speech recognition, [71, 41, 73] utilize the Hidden Markov Model (HMM) to link videos and actions. [66] initially produces uniform segmentations and iteratively refines boundaries by inserting repeated actions into the transcript. [43] introduces an alignment objective based on explicit context and length models, solvable via Viterbi, to generate pseudo labels for training a frame-wise classifier. Similarly, [42] and [70] propose novel learning objectives but still rely on Viterbi for optimal pseudo segmentation. Both [67, 69] use DTW to align videos to both ground-truth and negative transcripts, emphasizing the contrast between them. However, except for [66] , these methods require frame-by-frame calculations, making them inefficient. More recently, alignment-free methods have been introduced to enhance efficiency. [68] learns from the mutual consistency between frame-wise classification and category/length pairs of a segmentation. [44] enforces the output order of actions to match the transcript order using a novel loss function. Although POC [44] is primarily set-supervised, it can be extended to transcript supervision, making its results relevant for comparison.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.We implicitly assume St,i,j together with St,i, St,i,j \ St,i have non-zero measure. 


Decision: The paper presents a causal temporal representation learning framework for identifying distribution shifts and latent factors in nonstationary time series without strong prior knowledge of domain variables. It introduces a sparse transition assumption and provides theoretical identifiability results for both domain variables and latent causal processes from sequential observations. 

Strengths:

+ Tackles an important problem of causal temporal representation learning under nonstationarity with sparse transition assumptions than prior work

+ Provides identifiability results

+ Evaluates the method on both synthetic and realistic tasks (action segmentation)

Weaknesses:

+ Some key assumptions (e.g. weakly diverse lossy transitions) can be unrealistic 

+ Lacks comparison to relevant related work (that may rely on different identification assumptions)

Reviews: review: paper_summary: paper_summary: The paper introduces CtrlNS, a causal temporal representation learning framework based on a sparse transition assumption to identify distribution shifts without strong prior knowledge of the domain variables. Theoretical and experimental results show CtrlNS effectively identifies distribution shifts and latent factors, outperforming existing methods.
strength_weakness: strength_weakness: The authors argue that the paper derives identifiability results *“without prior knowledge of the domain variables”* (e.g., line 46, 61). However, this is an overstatement since CtrlNS still requires some prior knowledge or assumptions, such as the ground-truth number of different domains $U$. Moreover, while I understand it does not rely on Markov assumption of prior works (e.g., NCTRL), the proposed framework still requires a different set of assumptions such as weakly diverse lossy transition and mechanism sparsity. In fact, I am not sure whether the authors’ assumption can be claimed as “weaker” compared to prior works. Therefore, I feel those statements need to be toned down, and more discussion on the comparisons of those assumptions is suggested.
questions: questions: - It seems that the framework requires the prior knowledge of the number of environments $U$. This assumption should be made more explicit. Also, what happens under the misspecification of $U$?
- For the mathematical rigor, the statements need to take measure-zero sets into account. For example, I think the conditions $p(S_{t, i, j})>0$ (line 136), $p(S_{t, i})>0$ and $p(S_{t, i, j}\setminus S_{t, i})>0$ (line 141) should be included, just to name a few.
- The framework requires $U$ different transition networks, which could be a potential weakness, especially in terms of scalability. The discussion on this point would further strengthen the paper. For example, an efficient parameter sharing of networks [3] may alleviate such issues.
- The proposed work assumes no contemporaneous causal relationships. (I know this is a common assumption in this line of work) How can this assumption be relaxed? With the interventional data, is it possible for the proposed framework to generalize to the existence of instantaneous effects, maybe similar to iCITIRS [1]?
- A number of works in RL setting also consider the identifiability under the non-stationary process [2, 3]. For example, [3] considers the identifiability of both domain variables and causal structures. I suggest the authors include these related works in different areas.
- The utilization of mechanism sparsity for identifiability is also explored in [4]. How is it different from the sparse transition assumption in CtrlNS?

(minor)

- Style file seems to be different from the official NeurIPS template (e.g., fonts).

***References***

[1] Lippe, Phillip, et al. "Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems." *The Eleventh International Conference on Learning Representations*.

[2] Feng, Fan, et al. "Factored adaptation for non-stationary reinforcement learning." *Advances in Neural Information Processing Systems* 35 (2022): 31957-31971.

[3] Hwang, Inwoo, et al. "Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning." *International Conference on Machine Learning*. PMLR, 2024.

[4] Xu, Danru, et al. "A Sparsity Principle for Partially Observable Causal Representation Learning." *International Conference on Machine Learning*. PMLR, 2024.
limitations: limitations: .
paper_summary: paper_summary: The paper focuses on temporally causal representation learning, where the goal is to recover a latent causal process from nonstationary observation sequences. Existing works represent the source of the nonstationarity by either known domain variables or (autocorrelated) unknown domain variables with Markov structure. The paper proposes a sparsity based constraint on the latent transitions to achieve the identifiability of the domain variables. Coupled with the identified domain variable, they make use of the sufficient variability assumption to identify the latent causal process. Based on this theoretical framework, they introduce a sequential VAE based model, which encourages the independent noise variables and the sparse transitions. They evaluate (i) the performance on the latent process identifiability on a synthetic dataset following their model definition, and (ii) the usefulness of their model on a weakly-supervised video action segmentation task on real-world datasets of daily actions and cooking.
strength_weakness: strength_weakness: * As is common in identifiability literature, the assumptions of the theoretical framework do not seem to be testable on a real-world setup.
* Definition 6 (Weakly diverse lossy transitions) seems unrealistic and is not clear from the text: (i) The motivation for the “Lossy” is not clear to me, how is there a causal link between two variables, while changing the causal parent does not affect the effect variable? (ii) On the clarity of the definition: perhaps, some graphical model examples could help here. In addition, the assumption is not discussed in the main text, except stating in lines 227-228 that it is “a mild and realistic condition in the real-world scenarios, allowing for identical future latent states with differing past states”. Why is it mild and realistic?
* It is not clear to me why having domain variables without a Markov structure is a better model of the real-world environments than domain variables with Markov structure. I don’t see why environments in the real-world would swap at random multiple times along a sequence of time points. For example, in the provided real-world experiment, ground-truth actions in the video in Figure S5 occur in blocks and they are autocorrelated. The weak-supervision of the action order used in this experiment also supports their autocorrelation.
* According to the point above, the synthetic experimental setup seems unrealistic as it has randomly sampled domain variables.
* I think the first contribution statement in Lines 59-61 is ambiguous: "this is the first identifiability result that handles nonstationary time-delayed causally related latent temporal processes without prior knowledge of the domain variables". What is meant by "without the prior knowledge of domain variables" is a bit vague, while the difference with the existing work [Song+23] seems to be the assumptions on the temporal dependence of the domain variables. [Song+23] assumes temporally-dependent domain variables and predicts the unknown domain variables with an HMM, while this work assumes temporally independent domain variables and predicts them from sequential observations.

[Song+23]. Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. "Temporally disentangled representation learning under unknown nonstationarity" Neurips, 2023.
questions: questions: * Some assumptions are motivated in lines 219-224 as being aligned with human intuition: “separability states that if human observers cannot distinguish between two domains, it is unlikely that automated systems can achieve this distinction either. Secondly, variability requires that the transitions across domains are significant enough to be noticeable by humans, implying that there must be at least one altered edge in the causal graph across the domains.” I find it hard to follow this reasoning. Why are these the cases?
* For both experiments, architectural details are missing. Besides, some information on the synthetic experimental setups is missing: what are the functions $\mathbf{m}$ and $\mathbf{g}$ used?, What is the train/val/test split?, What parts of the experiment change with the changing seed?
* It is not clear to me what the model predicts on the action segmentation task. I think it predicts the action class per frame given the video frames and the action order. If so, why do you report MoF, IoU and IoD, instead of accuracy? How do you provide the weak-supervision information to the model? Why is the baseline NCTRL excluded from this task? Its generative model seems to fit the task well.
* It is not clear what the sparsity loss is. In line 266, it is said to be the L2 norm of the parameter in the transition estimation functions. Which parameter is this?
limitations: limitations: * The main limitation, the assumptions being untestable for a real-world setup, is not discussed in the main text.
paper_summary: paper_summary: The paper focuses on causal temporal representation learning for non-stationary time series. It adopts a sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. Based on the theoretical result, the authors introduce a novel framework, Causal Temporal Representation Learning with Nonstationary Sparse Transition (CtrlNS), designed to leverage the constraints on transition sparsity and conditional independence to reliably identify both distribution shifts and latent factors. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in recovering latent variables and domain indices.
strength_weakness: strength_weakness: 1.The authors lack discussion of some existing work, for example, the paper [1] also demonstrates identifiability by clustering nonstationary spatiotemporal data into different domains without prior knowledge. It is essential for authors to further discuss the similarities and differences between the two papers and alleviate my concerns about the novelty of this work.

2.In lines 259-261 of this paper, the authors mention ensuring the invertibility of the modeled mixing function through reconstruction loss, which is insufficient. Invertible Neural Networks (INNs) should be employed to ensure invertibility, as there will inevitably be errors in the reconstruction process.

[1] Zhao, Yu, et al. "Generative Causal Interpretation Model for Spatio-Temporal Representation Learning." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.
questions: questions: Please see the Weaknesses.
limitations: limitations: The authors have appropriately discussed the limitations and societal impacts of the model.
paper_summary: paper_summary: Most existing causal temporal representation learning models either assume the domain variables to be observable, or have a Markov prior over them. This paper first comes up an identifiability theory for sequential data affected by non-stationary latent causal processes under unknown distributional shifts. In particular, a novel causal temporal representation learning method is developed to identify the latent causal variables with time-delayed causal relations. Moreover, without relying on prior knowledge on distributional shifts, the domain variables can be inferred from sequential observations. The final experiments on both synthetic and real-world action segmentation data, demonstrate the superior performance of the novel method in estimating domain variables and latent causal processes.
strength_weakness: strength_weakness: Throughout the paper, I did not find the details about how to set up the parameters of the neural nets for the novel CTRL framework, which could be crucial for the others to pick up the work.
questions: questions: Except the action segmentation tasks, can the authors provide more real applications including healthcare and finance, as mentioned in the boarder impacts.
limitations: limitations: Yes, the paper discusses the potential negative societal impact of the work.

Comments: title: Additional experiment result on multiple random seeds.
comment: Dear Reviewer 3MKp,

We would like to express our sincere gratitude for the time and effort you invested in reviewing our submission. We greatly appreciate your acknowledgment of our contribution and the insightful comments you provided. We have carefully considered your feedback and have made the response, which we hope address your concerns. Should you have any further comments or additional suggestions, we would be more than happy to discuss them.

Thank you once again for your valuable input.

Best regards,

Authors of submission 4804
> It would be better to see the datasets generated from different seeds.

We include additional experimental results using three different random seeds to generate synthetic datasets. We compare our method with NCTRL and report the mean and standard deviation of the results across those three datasets in the following table. It can be observed that since NCTRL cannot generalize beyond the Markov assumption, its performance is weaker than our method and is also less stable in comparison.

| Method | MCC($z_t$)|Acc($u_t$)|
|--------|------|------|
|NCTRL|50.99$\pm$3.77|57.51$\pm$13.23|
|CtrlNS|95.04$\pm$1.39|97.09$\pm$0.89|

We sincerely appreciate the time and effort you dedicated to reviewing our submission and providing such insightful comments. If there are any unresolved concerns or additional thoughts, we would be more than happy to address them.

Best regards, 

Authors of submission 4804
Dear Reviewer 1QG3

We sincerely appreciate the time and effort you dedicated to reviewing our submission and providing such insightful comments. Your feedback is invaluable to us. If there are any unresolved concerns or additional thoughts, we would be more than happy to address them.

Thank you again for your thoughtful contributions.

Best regards,
Authors of submission 4804
Dear Reviewer 6hAo

We sincerely appreciate the time and effort you dedicated to reviewing our submission and providing such insightful comments. Your feedback is invaluable to us. If there are any unresolved concerns or additional thoughts, we would be more than happy to address them.

Thank you again for your thoughtful contributions.

Best regards,
Authors of submission 4804
Dear Reviewer Zo2n,

Thank you for providing your feedback. Please see our further response below:

> As far as I understand, the example provided, $B=Relu(A) + \epsilon_{B}$, does not satisfy the lossy assumption as clearly $\frac{\partial m_{b}}{\partial A} \neq 0$ everywhere. It is just $\frac{\partial m_{b}}{\partial A} = 0$ for $A\in (-inf,0)$. When $\frac{\partial m_{b}}{\partial A} = 0$ for all values of $A$, I still do not see how it can be a causal parent.

We would like to highlight the definition of the lossy assumption in line 136:

- Line 136: **There exists** an open set $S_{t,i,j}$ such that changing $z_{t−1,i}$ within this set will not change the value of $m_j$.

As we only require **there exists** a subset of $\mathcal{Z}$ where the partial derivative is zero, it indicates that we don't require "$\frac{\partial m_{b}}{\partial A} = 0$ for all values of $A$", hence this should not influence the existence of the causal link.

> For W3, your real-world dataset shows that a sequence generated from a temporally independent distribution is unrealistic. A realistic setup would be to test your model on only state sequences generated from a Markov chain.

We would like to note that:

1. The state sequence used in our experiment is not purely generated from a temporally independent distribution, and our method doesn't require the state sequence to be generated from a temporally independent distribution.
2. Our method can allow a mixture of independent and dependent settings (i.e., we can handle either dependent, independent, or both). Our synthetic experiment setup specifically explores this mixed case. It is clearly temporally dependent and can not be modeled by a Markov chain.

We respectfully disagree with the statement "using only state sequences generated from a Markov chain is a realistic setup" in our synthetic experiment for the following reasons:

1. Synthetic experiments are designed to validate the proposed theory and clearly distinguish it from existing baselines. Since we claim that our method achieves domain variable identifiability even when the data does not originate from a single Markov chain, it is essential that our synthetic experiments directly test this scenario. Specifically, in comparison to NCTRL, we demonstrate that NCTRL is unable to handle the more general case where $u_t$ is not generated from a Markov chain, whereas our method is capable of managing such complexity.

2. Regarding the setting mentioned by the reviewer, "only state sequences generated from a Markov chain," we understand that this is the synthetic experiment setup used in the NCTRL paper. We do not see the necessity for this setup in our synthetic experiments, as both NCTRL and our method are capable of handling this scenario and we are not claiming to outperform NCTRL in this scenario.

3. From the real-world experiments comparing NCTRL and our method, it is evident that our method outperforms NCTRL. This suggests that assuming the state sequence is generated from a Markov chain is not suitable for real-world settings.


> For Q2, architecture details are still missing.

We kindly ask the reviewer to specify which aspects of the architecture described in Section 4.1 remain unclear. We are more than happy to provide additional details and clarifications as needed.

> It would be better to see the datasets generated from different seeds.

We thank the reviewer for suggesting this additional experiment setting. We are in the process of expanding the experiments to include more datasets generated from multiple seeds. However, given it is close to the discussion deadline, we will do our utmost to include the results before the discussion concludes.

> The details on ground-truth transition and mixing function details are still missing.

The transition and mixing functions are implemented as randomly initialized MLPs with hidden dim of 32 with input and output dim 8. All implementation details can be found in our codebase provided in Appendix S3.3. Hope the attached code can make it clear.

> For Q4, the sparsity loss is wrong. L1 norm is commonly used to encourage sparsity. As this is the only architectural novelty, I am not sure how much the second contribution, (2) the CtrlNS framework, makes sense.

We would like to clarify that the most accurate way to enforce sparsity is through the $L_0$ norm. However, since calculating the gradient for $L_0$ is challenging, $L_p$ norms are commonly used as approximations. As indicated in line 1083 of Appendix S3.4, we tested both $L_1$ and $L_2$ norms and both work well in optimal case (MCC: 0.9690 vs 0.9704), and the setting we employed in the paper was selected for its superior stability. 

Regarding the contribution, we have already conducted an ablation study in the real-world experiment section. The results show that removing the sparsity loss leads to a performance drop, which supports the effectiveness of our proposed method.
Dear authors,

Thank you for taking time to answer my concerns, however, the following points are still unclear for me:
* For W2, 
  * As far as I understand, the example provided, $B = Relu(A) + \epsilon_B$, does not satisfy the lossy assumption as clearly $\frac{\partial m_B}{\partial A} \neq 0$ everywhere. It is just $\frac{\partial m_B}{\partial A} = 0$ for $A \in (- \inf, 0)$. When $\frac{\partial m_B}{\partial A} = 0$ for all values of $A$, I still do not see how it can be a causal parent.
* For W3, your real-world dataset shows that *a sequence generated from a temporally independent distribution* is unrealistic. A realistic setup would be to test your model on only state sequences generated from a Markov chain.
* For Q2, 
  * architecture details are still missing,
  * it would be better to see the datasets generated from different seeds,
  * the details on ground-truth transition and mixing function details are still missing,
* For Q4, the sparsity loss is wrong. L1 norm is commonly used to encourage sparsity. As this is the only architectural novelty, I am not sure how much the second contribution, (2) the CtrlNS framework, makes sense.