Causal Temporal Representation Learning with Nonstationary Sparse Transition
Authors: Xiangchen Song
Zijian Li
Guangyi Chen
Yujia Zheng
Yewen Fan
Xinshuai Dong
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal Representation Learning
Identifiability
Nonlinear ICA
Temporal Distribution Shift
Primary area: causal_inference

Abstract: Causal Temporal Representation Learning (Ctrl) methods aim to identify the temporal causal dynamics of complex nonstationary temporal sequences. Despite the success of existing Ctrl methods, they require either directly observing the domain variables or assuming a Markov prior on them. Such requirements limit the application of these methods in real-world scenarios when we do not have such prior knowledge of the domain variables. To address this problem, this work adopts a sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. In particular, we explore under what conditions on the significance of the variability of the transitions we can build a model to identify the distribution shifts. Based on the theoretical result, we introduce a novel framework, *Causal Temporal Representation Learning with Nonstationary Sparse Transition* (CtrlNS), designed to leverage the constraints on transition sparsity and conditional independence to reliably identify both distribution shifts and latent factors. Our experimental evaluations on synthetic and real-world datasets demonstrate significant improvements over existing baselines, highlighting the effectiveness of our approach.

Causal Temporal Representation Learning with Nonstationary Sparse Transition
Xiangchen Song'
Zijian Li
Guangyi Chen 1.2
Yujia Zheng'
Yewen Fan
Xinshuai Dong'
Kun Zhang
CCarnegie Mellon University
2 Mohamed bin Zayed University of Artificial Intelligence
fxiangchensong, kunz1) @duuueduuueeu edu
Abstract
Causal Temporal Representation Learning (Ctrl) methods aim to identify the tem- poral causal dynamics of complex nonstationary temporal sequences. Despite the success of existing Curl methods, they require either directly observing the domain variables or assuming a Markov prior on them. Such requirements limit the application of these methods in real-world scenarios when we do not have such prior knowledge of the domain variables. To address this problem, this work adopts sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. In particular, we ex- plore under what conditions the significance of the variability of the transitions we can build a model to identify the distribution shifts. Based on the theoretical result, introduce novel framework, Causal Temporal Representation Learning with Nonstationary Sparse Transition (CtrINS), designed to leverage the con- straints on transition sparsity and conditional independence reliably identify both distribution shifts and latent factors. Our experimental evaluations on synthetic and real-worte datasets demonstrate significant improvements over existing baselines, highlighting the effectiveness of our approach.
Introduction
Causal learning from sequential data remains fundamental yet challenging task [13].. Discovering temporal causal relations among observed variables has been extensively studied in the literature 44- [.. However, in many real-world scenarios such as video understanding [7], observed data are generated by causally related latent temporal processes confounders rather than direct causal edges. This leads to the task of causal temporal representation learning (Ctrl), which aims to build compact representations that concisely capture the data generation processes by inverting the mixing function that transforms latent factors into observations and identifying the transitions that govern the underlying latent causal dynamics. This learning problem is known to be challenging without specific assumptions g [9]. The task becomes significantly more complex with nonstationary transitions, which are often characterized by multiple distribution shifts across different domains, particularly when these domains or shifts are also unobserved.
Recent advances in unsupervised representation learning, particularly through nonlinear Indepen- dent Component Analysis (ICA), have shown promising results in identifying latent variables by incorporating side information such as class labels and domain indices [10-19]. For time-series data, historical information widely utilized to enhance the identifiability of latent temporal causal pro- cesses 23) However, existing studies primarily derive results under stationary conditions [1] 21
or nonstationary conditions with observed domain indices [13 22 23 These methods are limited in application general time series data typically nonstationary and domain information is difficult
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

have adopted Markov structure to handle nonstationary domain variables and can infer domain indices directly from observed data. (More related work can be found in Appendix (S4) However, these methods face significant limitations; some are inadequate for modeling time-delayed causal relationships latent spaces, and they rely on the Markov property, which cannot adequately capture the arbitrary nonstationary variations in domain variables. This leads us to the following important yet unresolved question:
to obtain. Recent studies [1 24 26
How can we establish identifiability of nonstationary nonlinear ICA for general sequence data without knowledge of the prior distribution of the domain variables?
Relying on observing domain variables or known Markov priors to capture nonstationarity seems counter-intuitive, especially considering how easily humans can identify domain shifts given sufficient variation on transitions, such as video action segmentation [27] 28] and recognition [29 tasks. In this work, we theoretically investigate the conditions on the significance of transition variability to identify distribution shifts. The core idea is transition clustering, assuming transitions within the same domain are similar, while transitions across different domains are distinct. Building on this identification theorem, we propose Causal Temporal Representation Learning with Nonstationary Sparse Transition (CtriNS), to identify both distribution shifts and latent temporal dynamics. Specifi- cally, we constrain the complexity of the transition function to identify domain shifts. Subsequently, with the identified domain variables, we learn the latent variables using conditional independence constraints. These two processes are jointly optimized within VAE framework.
The main contributions of this work are as follows: (1) To our best knowledge, this is the first identifiability result that handles nonstationary time-delayed causally related latent temporal processes without knowledge of the prior distribution of the domain variables. (2) We present CtrINS, a principled VAE-based framework for recovering both nonstationary domain variables and time- delayed latent causal dynamics. (3) Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in recovering latent variables and domain indices.
2 Problem Formulation
2.1 Nonstationary Time Series Generative Model
14
Me++
We first introduce nonstationary time-series generative model in our setting. The observational dataset is D = {x+} where x; E Rn" is produced from causally related, time-delayed latent components z, E IR" through an invertible mixing function g:
D.
#2
X = g(z))
(1)
R
In the nonstationary setting, transitions within the latent space vary over time. Define u as the domain or regime index variable, with u, corresponding domain variable at time step t. Assuming there are U distinct regimes, i.e., Uy E (1,2,..., U), each regime exhibits unknown distribution shifts. Those regimes different transition functions } which were originally in- troduced [32) through change factors to capture these distribution shifts in transition dynamics. The i-th component of latent variable ze, is then generated via i-th component of transition function m:
Figure I: Graphical model for nonstationary causally related time-delayed time-series data generation process with unob- served domain variables u4
characterized by U
2t,1 mm (ut, {ev. .f 2e E P.,,
(2)
where Pa(z),) represents the set of latent factors directly influencing zt,, which may include 1), E (1, For analytical simplicity, we
any subset of Zec {2r,5 T E 21, assume that the parents in the causal graph are restricted to elements in Z;-1. Extensions to higher- order cases, which involve multistep. time-delayed causal relations, are already discussed in Appendix S1.5 of [23]. These extensions are orthogonal to our contributions and are therefore omitted here for brevity. Importantly, in a nonstationary context, Pa(-) may also be sensitive to the domain index Wr. indicating that causal dependency graphs vary across different domains or regimes, which will be revisited in our later discussion on identifiability. We assume that the generation processes for each i-th component of 2; are mutually independent, given zac and u. Consistent with the existing

---

literature 223 251, we further assume that the noise terms ε,,i are independent both spatially and temporally. This assumption implies that there is no instantaneous causal influence among the latent causal processes. The graphical model corresponding to this setting is illustrated in Figure] |
2.2 Identifiability of Domain Variables and Latent Causal Processes
In this section we introduce the identifiability of both domain variables and time-delayed latent causal processes in Definitions and up to a permutation and component-wiss invertible transformations, then the latent causal relationships also immediately identifiable. This follows from the fact that conditional independence relations comprehensively characterize the time-delayed causal relations within a time-delayed causally sufficient system, in which there no latent causal confounders in the causal processes. Notably, invertible component- wise transformations on latent causal processes preserve their conditional independence relationships. We now present definitions concerning observational equivalence, the identifiability of domain variables and latent causal processes.
and respectively. If the estimated latent processes are identifiable at least
Definition 1 (Observational Equivalence). Formally, consider {x+ as a sequence of observed variables generated by true temporally causal latent processes specified by (m, u, p(E), g) given in Eqs. ① and (2). Here, m and denote the concalenated vector form across nt dimensions in the latent space. Similarly u for time steps to T. A learned generative model (ın,, ú, (((), g) is observationally equivalent to the ground truth one (m, u, e))), g) if the model distribution p (x1))=)) matches the data distribution Pm, u.p.R ((x))==) everywhere.
Definition 2 (Identifiable Domain Variables). Domain variables are said to be identifiable up to label swapping if observational equivalence (Def. 7 implies identifiability of domain variables up to a permutation a for domain indices:
i.....P. goc.g(((())) ((x))== Pm (
(x+ 12= = i,
Definition 3 (Identifiable Latent Causal Processes). The latent causal processes are said to be identifiable if observational equivalence (Def. [ leads the identifiability of latent variables up to permutation л and component-wiss invertible transformation T
fr &e (xx)TT=))= Pm.... & ((xx) + 8 (xx)) T ToTTTO ooTo- Vx, E Ex (4)
where X denotes the observation space.
3 Identifiability Theory
In this section, demonstrate that under mild conditions, the domain variables u, identifiable up to label swapping and the latent variables 2, are identifiable up to permutation and component-wise transformations. We partition our theoretical discussion into two sections: (1) identifiability of nonstationary discrete domain variables u, and (2) identifiability of latent causal processes. We slightly extend the usage of supp(-) )) to define the square matrix support and the support of square matrix function as follows:
Definition (Matrix Support). The support (set) of a square matrix A E RRxx is defined using the indices of non-zero entries as:
supp(A) {(i,5) Aij + 0}.
(5)
Definition 5 (Matrix Function Support). The support (set) of square matrix A A -> R is defined as:
supp(A(()) {(i.)) 30 e e, A(0)s, + #0.
(6)
For brevity, let M and M denote the binary matrices representing the support of the Jacobian J (zz) and Jm (2:). respectively. The (i, jtth entry of M is 1 if and only if (i,j) E supp(Jm). We further define the transition complexity using its Fréchet norm as |M - Mij. and similarly
for M. In the nonstationary setting, this support matrix becomes a function of the domain index u. denoted as M,, and Mu. Additionally, we introduce the concept of weakly diverse lossy transitions for the data generation process, which is formally defined below:
3

---

Definition 6 (Weakly Diverse Lossy Transition). The set of transition functions described in Eq. said to be diverse lossy if satisfies the following conditions:
1. (Lossy) For every time and indices tuple (t, (.) with edge 21-1, + 21,3 representing causal link defined with the parents set Pa(zt,) in Eq. 2 transition function m, is lossy
transformation w.r.t. Zt-1,s i.e. there exists an open St. changing 2t-1, within this set will not change the value m. i.e. V2-1, E St 12i-14
2. (Weakly Diverse) For every element 21-1,5 of the latent variable Zi-1 and its correspond- ing children set Tt,; = [j 1t-1, E Pa(zt.). E (1,2.....n n..,, }. transition functions {m;]ieJn, are weakly diverse iee, the intersection of the sets St njeJ,, is not empty, and such sets are diverse, i.e. St + O, and Sta.s St. + 0, Vj Jr.
3.1 Identifiability of Domain Variables
Theorem 1 (Identifiability of Domain Variables). Suppose that the dataset 'D are generated from the nonstationary data generation process as described in Eqs. ① and @ Suppose the transitions are weakly diverse lossy (Def. 6 and the following assumptions hold:
(Mechanism Separability) There exists a ground truth mapping C : X X -+ U determined the real domain indices, i.e., u, = C(xx-1,,,))
it. (Mechanism Sparsity) The estimated transition complexity on dataset 'D is less than or equal to ground truth transition complexity, i.e., Ep Ep|Mal S Ep|Mul.
ii. (Mechanism Variability) Mechanisms are sufficiently different. For all u + u'. Mu # M, i.e. there exists index (i.j) such that [Mules * [Murle
Then the domain variables 4e is identifiable up to label swapping (Def. 2
Theorem states that we successfully learn a set of estimated transitions {m Juz the decoder g.
and the domain clustering assignment é, where m. corresponds to the estimation of Eq. (2) for particular regime or domain and the system can fit the data as follows:
R g o más (xx-1) and à = x(x.
(7)
assuming that the transition complexity kept low (as per Assumption mi. Then the estimated domain variables de must be the true domain variables de up to permutation.
Proof sketch The core idea of this proof is to demonstrate that the global minimum of transition complexity can only be achieved when the domain variables ut are correctly estimated. (1) First, consider the case when we have optimal decoder estimation g' which is component- transformation of the ground truth, incorrect estimations of u, will strictly increase the transition
complexity, i.e., Ep|M > Epl Mol (2) Second, we show that with arbitrary estimations ú,, the transition complexity for any non-optimal decoder estimation g will be equal to or higher than that for the optimal g" i.e., Ep Mal 2 Ep MMl.- Thus, the global minimum of transition complexity can only be achieved when u, is optimally estimated, which must be permuted version of the ground truth domain variables un A comprehensive proof can be found in Appendix SI.I
3.2 Remark on Mechanism Variability
The assumption of mechanism variability, as stated in Assumption ffi requires that the Jacobian support matrices differ across domains, indicating that the causal graph connecting past states (Z:-1) to current states (2,) must differ by at least one edge. Addressing scenarios where the causal graphs remain identical but the transition functions associated with the edges vary is generally challenging without imposing additional assumptions. more detailed discussion of the difficulties involved in such cases is provided in Appendix $1.4.4 To effectively address these scenarios, we extend the concept of the Jacobian support matrix by incorporating higher-order derivatives. This extension
We implicitly assume Se,i together with St,s, Se.i. \ St.s have non-zero measure.

---

provides more detailed characterization of the variability in transition functions across different domains. We now present the following definition to formalize this concept: Definition 7 (Higher Order Partial Derivative Support Matrix). The k-th order partial derivative support matrix for transition m denoted as M is binary
matrix with
[ =1 =
= Iz E Z,
8"
+ 0.
(8)
We utilize the variability in the higher-order partial derivative support matrix to extend the identifia- bility results of Theorem | This extension applies to cases where the causal graphs remain identical across two domains, yet the transition functions take different forms.
Corollary 1 (Identifiability under Function Variability). Suppose the data 'D is generated from the nonstationary data generation process described in (I) and (2). Assume the transitions are weakly diverse lossy (Def. O and the mechanism separability cnammpttinaalona with the following assumptions hold:
V. (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K E N such that for all u + u', there exists k S K. M # My i.e. there exists index (i,j) such that [Mb]] t.j + [M
vi. (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset D no more than ground truth transition complexity,
ED EMMIIES
ED
M
(9)
Then the domain variables un are identifiable up to label swapping (Def. 2
To prove this corollary, we leverage the property that, for any two distinct domains, there exists an edge in the causal graph such that the supports of their k-th order partial derivatives differ. This difference ensures the separability of the two domains. A detailed proof can be found in Appendix S1.2
3.3 Identifiability of Latent Causal Process
Once the identifiability of u, is achieved, the problem reduces nonstationary temporal nonlinear ICA with observed domain index. Leveraging the sufficient variability approach proposed in (23),
we demonstrate full identifiability of the data generation process. This sufficient variability concept
is further incorporated into the following lemma, adapted from Theorem 2 in (23):
Lemma 1 (Theorem 2 in Yao et al., (3) Suppose that the data 'D are generated from the nonsta- tionary data generation process as described in Eqs. (T) and (2). Let (k((u) denote the logarithmic density of k-th variable in Zi, i.e., (u) 2 log p(zz, (1,1, u), and there exists an invertible anvettiblefunction
g that maps x, to 2t, i.e., 2, g(x)) such that the components of 2s are mutually independent conditional on 2.-1- (Sufficient variability) Let
(m)
(n)(u))
pna())) )
Vk,e(u) A
(10)
0211021-111 021,021-112
03 8e (1)
vx,c(u) 4
(11)
Oz2 02-1111
02
... Oz 020-1
02 (2)
(1)
mml(l)
2 (U -1)
Ske (vac(1)T+ ; (U)
(12)
027
027,
Ozi (U)((U)
0zi
@mp((1)
@maaU -1-) -
Skt (V. Ve (U))
(13)
Suppose X₁ = g(z)) and that the conditional distribution p(Zk, Zi-1) may change across m domains. Suppose that the components of 2 are mutually independent conditional on Z in each context. Assume that the components of 2t produced by g are also mutually independent conditional
on 2- If the 2n function vectors Sk,t and $5,ts with 1, 1.2.... n, are linearly independent, then 2; is permuted invertible component-win wise transformation of zz-
5

---

Then, in conjunction with Theorem complete identifiability is achieved for both the domain variables u, and the independent components z, See detailed proof in Appendix [SI.3
Theorem (Identifiability of the Latent Causal Processes). Suppose that the observed dataset 'D is generated from the nonstationary data generation process as described in Eqs. ① and (2). which satisfies the conditions in both Theorem and Lemma then the domain variables un are identifiable up to label swapping (Def. 2 and latent causal process are identifiable up to permutation and component-we transformation (Def.
Discussion on Assumptions The proof of Theorem I relies several essential assumptions that correspond with human intuition regarding domain transitions. First, the assumption of separability posits that if human observers unable to differentiate between two domains, it is improbable that automated systems will achieve such a distinction. Second, the variability assumption requires that the differences in transitions between domains be substantial enough to be perceptible to humans. This often results in changes to the temporal causal structure across domains, indicating that at least one edge in the causal graph must differ between the domains.
The mechanism sparsity is a standard assumption that has been previously explored in [33 19 [8]
using sparsity regularization to enforce the sparsity of the estimated function. The assumption of weakly diverse lossy transitions a mild and realistic condition in real-world scenarios, allowing for identical future latent states with differing past states. The sufficient variability in Theorem 2 widely
explored and adopted in nonlinear ICA literature [2 22 23, 25 26 For more detailed discussion of the feasibility and intuition behind these assumptions, we refer the reader to the Appendix (S1.4)
The CtrINS Framework
4.1 Model Architecture
architecture, incor- porating dedicate modules to handle nonstationarity. It en- forces the conditions discussed in Sec.Blas constraints. As shown in Fig. 2 the framework consists of three primary components: (I) Sparse Transition, (2) Prior Network, and (3) Encoder-Decode
Our framework builds VAE 34 35
fiz ug..... ur
Sparse Transition {m
Prior Network
mn
(ūe, 2e, &e 1)
Sparse
(2) Lg. ..... 2r)
LKLD
Sparse Transition The transition module in our frame- work designed to estimate transition functions {mn.)) ) and a clustering function Ć as specified in Eq. (7). As high- lighted in Sec. 3 the primary objective of this module is to model the transitions in the latent space and minimize the empirical transition complexity. To achieve this, we implemented U different transition networks for various ((,,, -) and added sparsity regularization the transition functions via a sparsity loss. A gating function with a (har)--Guubel--Sfttma function was used to generate ür which was then employed to select the corresponding transition network ma. This network was further used to calculate the transition loss, which is explained in detail in Sec.
Encoder t
Decoder
CRsson
{x1, X2...., .y)
{ 82
Figure 2: Illustration of CtrINS with (1) Sparse Transition, (2) Prior Network, (3) Seccoder-Decoder Module.
Prior Network The Prior Network module aims to effectively estimate the prior distribution p(2r, 22,11 a.). This is achieved by evaluating p(&,| 2 à.) = P., " 2 2)-1))
Dm
as..
where m. (ā., is the learned holistic inverse dynamics model. To ensure the conditional indepen- dence of the estimated latent variables, p(ž, 22,1).. we utilize an isomorphic noise distribution for e and aggregate all estimated component densities to obtain the joint distribution p(2. 2;-1, ux) as
shown in Eq. (14). Given the lower-triangular nature of the Jacobian, its determinant can be computed
the product of its diagonal terms. Detailed derivations is provided in Appendix S3.1
Omn
log p (ă: 1ē it)
log p(é 0z) +
log
(14)
Conditicnal indopendense
Lowes-tiangglar Jacobiua

---

Encoder-Decoder The third component is an Decoodr- module that utilizes reconstruction loss enforce the invertibility of the learned mixing function g. Specifically, the encoder fits the demixing function g and the decoder fits the mixing function
4.2
Optimization
The first training objective of CtriNS is to fit the estimated transitions with minimum transition complexity according Eq. 7
Expase
EpL(ma (21-1) 21)
BD Mal
(15)
Transition loss
Sparsity loss.
where I(.,-) a regression loss function to fit the transition estimations, and the sparsity loss approximated via L2 norm of the parameter in the transition estimation functions.
Then the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written follows (complete derivation steps in Appendix S3.2h:
ELBO Ez,
log Paum (xt Zc)
log (Ze Z:-1; uc) Σ
log (z: x:)
(16)
Cioon
CRID
We use mean-squared error for the reconstruction likelihood loss Recon The KL divergence LKLD is estimated via sampling approach since with learned nonparametric transition prior, the distribution does not have an explicit form. Specifically, we obtain the log-likelihood of the posterior, evaluate the prior log p (2: 1 ût) in Eq. (14). and compute their mean difference in the dataset the KL. loss: LKLD (a. log g (2, log (Z, ûc).
Experiments
We assessed the identifiability performance of CtrINS on both synthetic and real-world datasets. For synthetic datasets, where we control the data generation process completely, we conducted a comprehensive evaluation. This evaluation covers the full spectrum of unknown nonstationary causal temporal representation learning, including metrics for both domain variables and the latent causal processes. In real-world scenarios, CtrINS employed in video action segmentation tasks. The evaluation metrics focus on the accuracy of action estimation for each video frame, which reflects the identifiability of domain variables.
5.1 Synthetic Experiments Causal Representation Learning
Evaluation Metrics For domain variables, we assessed the clustering accuracy (Acc) estimate discrete domain variables u, As the label order in clustering algorithms is not predetermined, we selected the order that yielded the highest accuracy score. For the latent causal processes, we computed the mean correlation coefficient (MCC) between the estimated latent variables 2, and the ground truth 2,- The MCC, a standard measure in the ICA literature for continuous variables, assesses the identifiability of the learned latent causal processes. We adjusted the reported MCC values Table by multiplying them by 100 to enhance the significance of the comparisons.
Baselines We compared our method with identifiable nonlinear ICA methods: (1) Beta BetaVAE (36], which ignores both history and nonstationarity information. (2) i-VAE [13] and TCL [0), which leverage nonstationarity to establish identifiability but assume independent factors. (3) Slow VAF 21 and PCL (Л]. which exploit temporal constraints but assume independent sources and stationary processes. (4) TDRL (23]. which assumes nonstationary causal processes but with observed domain indices. (5) HMNLICA [15]. which considers the unobserved nonstationary part in the data generation process but does not allow any causally related time-delayed relations. (6) NCTRL (25), which extends HMNLICA to an autoregressive setting to allow causally related time-delayed relations in the latent space but still assumes Markov chain the domain variables.
Result and Analysis We generate synthetic datasets that satisfy our identifiability conditions in Theorems and detailed procedures are in Appendix S2.1 The primary findings are presented in Table T Note: the MCC metric consistently available in all methods; however, the Acc metric for ue is only applicable to methods capable of estimating domain variables ut-

---

Table I: Experiment results of synthetic dataset on baseline models and the proposed CtrINS. All experiments were conducted using three different random seeds to calculate the average and standard deviation. The best results are highlighted in bold.
Up
Method
2, MCC
Acc (%)
Ground Truth TDRL(GT)
96.93 + 0.16
TCL
24.19 + 0.85
PCL
38.46 + 6.85
Beta VAE
42.37 + 1.47
N/A
SIowVAE
41.82 + 2.55
i-VAE
81.60 + 2.51
TDRL
53.45 + 1.31
HMNLICA
17.82 + 30.87
13.67 - 23.67
Estimated
NCTRL
47.27 2.15
34.94 + 4.20
CtrINS
96.74 + 0.17
98.21 0.05
In the first row of Table L we evaluated a recent nonlinear temporal ICA method, TDRL, providing ground truth u, to establish an upper performance limit for the proposed framework. The high MCC (> 0.95) indicates the model's identifiability. Subsequently, the table lists six baseline methods that neglect the nonstationary domain variables, with none achieving a high MCC. The remaining approaches, including proposed CtrINS, are able to estimate the domain variables u, and recover the latent variables. In particular, HMNLICA exhibits instability during training, leading considerable performance variability. This instability stems from HMNLICA's inability allow time-delayed causal relationships among hidden variables 2,, leading to model training failure when the actual domain variables deviate from the Markov assumption. In contrast, NCTRL, which extends TDRL under the same assumption, demonstrates enhanced stability and performance over HMNLICA by accommodating transitions in Z,. However, since they use incorrect assumption on the nonstationary domain variables, the performance of those methods can be even worse than methods which do not include the domain information. Nevertheless, considering the significant nonstationarity and deviation from the Markov properties, those methods struggled to robustly estimate either the domain variables or the latent causal processes. Compared to all baselines, our proposed CtrINS reliably recovers both (MCC> 0.95) and 2, (Ace 95%), and the MCC is on par with the upper performance bound when domain variables are given, justifying effectivess.
Detailed Training Analysis To further validate our the-
100
Uy Acc
oretical analysis, we present visualization of the entire training process for CtrINS in Figure 3 It consists of three phases: (1) Phase 1, the initial estimations for both ut (2) During Phase 2, the accuracy of the estimation of ue continues the quality of the estimation of z, remains relatively un- changed compared with Phase 1. (3) In Phase 3, as u, becomes clearly identifiable, the MCC of z, progressively improves, ultimately achieving full identifiability. This three-phase process aligns perfectly with theoretical predictions. According to Theorem [ phases and should exhibit suboptimal z, estimations, while sparsity constraints can still guide training and improve the accu- racy for domain variables uy. Once the accuracy of 44 approaches high, Theorem D drives the improvement in MCC for z, estimations, leading to the final achievement of full identifiability of both latent causal processes for z, and domain variables u4-
MCC
:hasse
50
and Z, are imprecise.
Phase 3:
improve, although
Phase
40
10'
10a
Figure 3: Visualization of three phase training process of CtrINS.
5.2 Real-world Application on Weakly Supervised Action Segmentation
Experiment Setup Our method was tested on the video action segmentation task to estimate actions (domain variables wz). Following (37 28 we use the same weakly supervised setting utilizing meta-information, such as action order. The evaluation included several metrics: Mean-over-Frame (MoF), the percentage of correctly predicted labels per frame; Inteersection-over--nnii (IoU), defined

---

Table 2: Real-world experiment result action segmentation task. We the reported value for the baseline methods from 28. Best results are highlighted in bold.
Dataset
Method
MoF
IoU
IoD
HMM+RNN
41
11.9
CDFL
45.0
19.5
25.8
TASL 40
42.1
23.3
33
Hollywood
MuCon 37
13.9
ATBA 28
47.7
28.5
44.9
CtrINS (Ours)
52.9 13.1
32.7
52.4
NN-Viterbi 43
26.5
10.7
24.0
CDFL 42
31.9
11.5
23.8
TASL 40
40.7
14.5
25.1
CrossTask
POC
44
42.8
15.6
ATBA
28
50.6
15.7
24.6
CtrINS (Ours)
54.0
15.7:0.
23.6408
as INMI//TTM and Intersection-ove---etteccii I and / are the ground-truth segment and the predicted segment with the same class.
Datasets Our evaluation used two datasets: Hollywood Extended [38] which includes 937 videos with 16 daily action categories, and CrossTask [39]. focusing on 14 of 18 primary tasks related to cooking comprising 2552 videos across 80 action categories.
Model Design Our model build on top of ATBA (28] method which uses multi-layer transformers backbone networks. We add our sparse transition module with the sparsity loss function detailed in Sec. 4.2 Specifically, we integrated a temporally latent transition layer into ATBA's backbone, using transformer layer across time axis for the Hollywood dataset and LSTM for the CrossTase dataset. To encourage sparsity in the latent transitions, Le regularization is applied to the weights of the temporally latent transition layer.
Result and Analysis The primary outcomes for real-world applications in action segmentation summarized in Table Traditional methods based on hidden Markov models, such as HMM+RNN [1 and NN-Viterbi [43] face challenges in these real- world scenarios. This observation corroborates previous discussions on the limitations of earlier iden- tifiability methods [15] 22 (25). which depend on the Markov assumption for domain variables. Our ap- proach significantly outperforms the baselines in both the Hollywood and Cross CrossTask Task datasets across most metrics. Especially in the Hollywood dataset, our method outperforms the base ATBA model by quite a large margin. Notably, the Mean-over Frrer-rrmes (MoF) metric aligns well with our identifiability results for domain variables uy- Our method demonstrates sub- stantial superiority in this metric. For Intersection- over-Union (IoU) and Intersection-over: Deve-Deeecttio (IoD), results are comparable to those of the base- line methods the CrossTask dataset and show its superiority in the Hollywood dataset. Furthermore, our proposed sparse transition module which aligns with human intuition and is easily integrated into ex- isting methods like a plug-in in module, thus further enhancing its impact in real-world scenarios.
Grund Butt
ATBA
Ground Tutt
ATDA
Ous
Figure 4: Two illustrative visualizations of the action segmentation task on the Hollywood dataset are presented. The colors represent the ground truth and the predicted action labels for each frame, as produced by the baseline ATBA and our proposed CtrINS.
To make the illustration more straightforward, some example segmentation results from the Hol- By comparing the number of distinct actions and the
lywood dataset are visualized in Figure 4

---

segmentation boundaries between our method and the baseline, it is evident that our CtrINS estimates the actions more accurately, demonstrating improved performance.
Table 3: Ablation study on sparse transition
Abalation Study Furthermone, we conducted an ablation study on the sparse transition module, as detailed in Table In this study, we test on a subset of Hollywood dataset for computational efficiency. For methods we compared, "- Complexity" refers to the configuration where we retain the latent transi- tion layers but omit the sparse transition complexity regularization term from these layers, and "- Module" indicates the removal of the entire sparse transition module, effectively reverting the model to the baseline ATBA model. The comparative results in Table demonstrate that both the dedicated design of the sparse transition module and the complexity regularization term enhance the performance.
module in Hollywood dataset.
Method
MoF
IoU
loD
CtrINS
52.9
32.7
52.4
CComplexity
50.5
31.5
51.5
-Module
47.7
28.5
44.9
6 Conclusion
In this study, we developed a comprehensive identifiability theory tailored for general sequential data influenced by nonstationary causal processes under unspecified distributional changes. We then introduced CtrINS, a principled approach to recover both latent causal variables with their time- delayed causal relations, as well determining the values of domain variables from observational data without relying on distributional or structural prior knowledge. Our experimental results demonstrate that CtrINS can reliably estimate the domain indices and recover the latent causal process. And such module can be easily adapted to handle real-world scenarios such as action segmentation task.
7 Limitations
As noted in Sec. 3.2, our main theorem relies on the condition that causal graphs among different domains must be distinct. Although our experiments indicate that this assumption is generally sufficient, there are scenarios in which it may not hold, meaning that the transition causal graphs identical for two different domains, but the actual transition functions are different. We have addressed this partially through an extension to the mechanism variability assumption to higher-order cases (Corollary D However, dealing with situations where transition graphs remain the same across all higher orders remains a challenge. We acknowledge this as a limitation and suggest it as an area for future exploration. We also observed that the random initialization of the nonlinear ICA framework can influence the total number of epochs needed to achieve identifiability, as illustrated in Figure 3 Also, for the computational efficiency, the TDRL framework we adopted involves prior network that calculated each dimension in the latent space one by one, thus making the training efficiency suboptimal. Since this is not directly related major claim which our sparse transition design, we acknowledge this as limitation and leave it for future work.
8 Boarder Impacts
This work proposes a theoretical analysis and technical methods to learn the causal representation from time-series data, which facilitate the construction of more transparent and interpretable models to understand the causal effect in the real world. This could be beneficial in a variety of sectors, including healthcare, finance, and technology. In contrast, misinterpretations of causal relationships could also have significant negative implications in these fields, which must be carefully done to avoid unfair or biased predictions.
Acknowledgment
The authors would like to thank the anonymous reviewers for helpful comments and suggestions during the reviewing process. The authors would also like to acknowledge the support from NSF Award No. 2229881, Al Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract 19151, and grants from Salesforce, Apple Inc., Quris AI, and Florin Court Capital.
10

---

References
[1] Carlo Berzuini, Philip Dawid, and Luisa Bernardinell. Causality: Statistical perspectives and applications. John Wiley & Sons, 2012.
[2] Eric Ghysels, Jonathan B Hill, and Kaiji Motegi. Testing for granger causality with mixed frequency data. Journal of Econometrics, 192(1):207-230, 2016.
[3] Karl Friston. Causal modelling and brain connectivity in functional magnetic resonance imaging. PLoS biology, 7(2):e1000033 2009.
[4] Clive WJ Granger. Testing for causality: A personal viewpoint. Journal of Economic Dynamics and control, 2:329-355 352, 1980.
[5] Mingming Gong, Kun Zhang, Bernhard Schoelkopf, Dacheng Tao, and Philipp Geiger. Discov- ering temporal causal relations from subsampled data. International Conference on Machine Learning, pages 1898-1906. PMLR, 2015.
[6] Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of a structural vector autoregression model using non- gaussianity. Journal of Machine Learning Research, 11 (5), 2010.
[7] Nadine Behrmann, Mohsen Fayyaz, Juergen Gall, and Mehdi Noroozi. Long short view feature decomposition via contrastive video representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9244-9253, 2021.
[8] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124. PMLR, 2019.
[9] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, (2(3):4294439 1999.
[10] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
[11] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[12] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[13] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statisties, pages 2207-2217, PMLR, 2020.
[14] Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-flow networks (gin). arXiv preprint arXiv:2001 04487, 2020.
[15] Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-948, PMLR, 2020.
[16] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pages 1145-114472 PMI.R, 17-23 Jul 2022. URL https: btttss:/preee http::////cceee.e../m/m mlr prwss/ wwwps//wiee.... http//111////// https//v12kkkooogg22 html

---

[17] Sebastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi LE PRIOL, Alexan- dre Lacoste, and Simon Lacoste-Julien Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2022. URL httpss: het.://w.eeetooorrmrdddaaaaaaaaa
[18] Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Nonparametrie partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint AxXiv:2401.04800 2024.
[19] Yujia Zheng and Kun Zhang. Generalizing nonlinear ICA beyond structural sparsity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
/ /ooeeneview mett//foruuriit-ttgg10gggss
[20] Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica arXiv preprint arXiv:2106. 09620, 2021.
[21] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint wiXi:220771199300 2020.
[22] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Representations, 2022. URL https:///penrrevee: wtwp:arrwiieeneeeefrrii... F.D
[23] Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. Advances in Neural Information Processing Systems, 22226222225503 2022.
[24] Carles Baisells-Rodas, Yixin Wang, and Yingzhen Li. On the identifiability of switching dynamical systems. In Forty-first International Conference on Machine Learning, 2024.
[25] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. In Seirth-eeventh Conference Neural Information Processing Systems, 2023.
URL httppppp///peeeeeeiee wwt.tnernnumn..e. idd wiw.orrmmniVVVCCCLLL... V8CHCGYLKf.
[26] Zijian Li, Ruichu Cai, Zhenhui Yang, Haiqin Huang, Guangyi Chen, Yifan Shen, Zhengming Chen, Xiangchen Song, Zhifeng Hao, and Kun Zhang. When and how: Learning identifiable latent states for nonstationary time series forecasting, 2024.
[27] John Ridley, Huseyin Coskun, David Joseph Tan, Nassir Navab, and Federico Tombari. Trans- formers in action: weakly supervised action segmentation. arXiv preprint arXiv:2201.0567 2022.
[28] Anghhi Xu and Wei-Shi Zheng. Efficient and effective weakly-supervised action segmentation via iilioon-taanition--aa boundary alignment. arXiv preprint arXiv:2403. 19225, 2024.
[29] Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierarchical deep temporal model for group activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971-1980, 2016.
[30] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees GM Snoek. group activity recognition. In Proceedings of the IEEF/CVF Conference on computer vision and pattern recognition, pages 839-848, 2020.
Actor-trance trans-ornnfrrss for
[31] Dongkeun Kim, Jinsung Lee, Minsu Cho, and Suha Kwak. Detector-free weakly supervised group activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20083-20093, 2022.
[32] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. AdaRL: What, where, and how to adapt transfer reinforcement learning. In International Conference on Learning Representations, 2022. URL
12

---

[33] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neural Information Processing Systems, 2022. URL. httts:://operreviiiii net/ forum?2d--N
[34] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprini 1312v21312.611 14, 2013.
[35] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. International Conference Machine Leurning, pages 5670-5679. PMLR, 2018
[36] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, wtt.://www.........oooo.tssssrrr
2017. URL
[37] Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, and Juergen Gall. Fast Weakly Supervised Action Segmentation Using Mutual Consistency, PAMI, 2021.
[38] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly supervised action labeling in videos under ordering constraints. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 628-643. Springer, 2014.
[39] Dimitri Zhukov, Jean- CaantBaptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545, 2019.
[40] Zijia Lu and Ehsan Elhamifar. Weakly-supervised action segmentation and alignment via transcript-awant cnioo-of-subspacee learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
October 2021.
[41] Alexander Richard, Hilde Kuchne, and Juergen Gall. Weakly supervised action learning with based fine-to-conee modeling. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 754-763, 2017.
[42] Jun Li, Peng Lei, and Sinisa Todorovic. Weakly supervised energy- based learning for action segmentation. In Proceedings of the IEEE/CVÉ international conference on computer vision, pages
2019.
[43] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnttwork-iierri:: A framework for weakly supervised video learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7386-7395, 2018
[44] Zijia Lu and Ehsan Elhamifar. Set-supervised action learning in procedural task videos via pairwise order consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19903-19913, 2022.
[45] Inwoo Hwang, Yunhyeok Kwak, Suhyung Choi, Byoung- Tak Zhang, and Sanghack Lee. Quan- tized local independence discovery for fine-grained causal dynamics learning in reinforcement learning, 2024. URL http:/tttt::/o ntttforum? -UGAdd--UUUUJAbb
[46] Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning, International Conference on Machine Learning (ICML), 2024.
[47] Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multiple distributions: A general setting, 2024.
[48] Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(2), 2006.
13

---

[49] Erich Kummerfeld and Joseph Ramsey. Causal clustering for 1-factor measurement models. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1655-1664, 2016.
[50] Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 15:5549-5561, 2022.
[51] Charles Spearman. Pearson's contribution to the theory of two factors. British Journal of Psychology, 19(1):95, 1928.
[52] Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neurocomputing, 72(7-9):0024-2222 2009.
[53] Ruichu Cai and Feng Xie. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 2019.
[54] Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. General- ized independent noise condition for estimating latent variable causal graphs. arXiv preprint arXiv:2010.04917, 2020.
[55] Feng Xie, Biwei Huang. Zhengming Chen, Yangbo Hc, Zhi Geng, and Kun Zhang. Identification of linear non- non-gaussian latent hierarchical structure. In International Conference Machine Learning, pages 24370-24387. PMLR, 2022.
[56] Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
[57] Nevin L Zhang. Hierarchical latent class models for cluster analysis. The Journal of Machine Learning Research, 5:697-723, 2004.
[58) Myung Jin Choi, Vincent YF Tan, Animashree Anandkumar, and Alan Willsky. Learning latent tree graphical models. Journal of Machine Learning Research, 12:1771-1812, 2011.
[59] Mathias Drton, Shaowei Lin, Luca Weihs, and Piotr Zwiernik. Marginal likelihood and model selection for gaussian latent tree and forest models, 2015.
[60] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. CITRIS: Causal identifiability from temporal intervened sequences. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022. URL https://openrevier het./forummtd--HHTTH.H...
[61] Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevi.c www. wwtps/nnneeooiiioeeeeeeeeee ftr..etee/orrrrmo.o....oggo
[62] Yu Zhao, Pan Deng, Junting Liu, Xiaofeng Jia, and Jianwei Zhang. Generative causal inter- pretation model for spatio-temporal representation learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3537-3548, 2023.
[63] Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process, 2024.
[64] Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non- stationary reinforcement learning. Advances in Neural Information Processing Systems, 35: 31957-31971, 2022.
[65] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal modeling for weakly supervised action laheling. In Proceedings of the European Conference on Computer Vision, pages 137-153. Springer, 2016.
14

---

[66] Li Ding and Chenliang Xu. Weakly-supervised action segmentation with iterative soft boundary assignment. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 6508-65 -6066666, 2018. 8.
[67] Chien-Yi Chang. De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3546-3555, 2019.
[68] Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, and Juergen Gall. Fast weakly supervised action segmentation using mutual consistency. IEEE Transactions Pattern Analysis and Machine Intelligence, 44( (410))61966628,
96-6208, 2021.
[69] Xiaobin Chang, Frederick Tung, and Greg Mori. Learning discriminative prototypes with dynamic time warping. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8395-8444 8404, 2021.
[70] Zijia Lu and Ehsan Elhamifar. Weakly superi-d action segmentation and alignment via transcript-aware union-of-subspaces learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8085-8095, 2021.
[71] Hilde Kuehne, Alexander Richard, and Juergen Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding, 163:78 89, 2017.
[72] Yaser Souri, Yazan Abu Farha, Fabien Despinoy, Gianpiero Francesca, and Juergen Gall. Fifa: Fast inference approximation for action segmentation. Proceedings of the DAGM German Conference on Pattern Recognition, pages 282-296. Springer, 2022.
[73] Hilde Kuehne, Alexander Richard, and Juergen Gall. A hybrid rnn-hmm approach for weakly supervised temporal action segmentation. IEEE transactions on pattern analysis and machine intelligence, 42(4):765-779, 2018.
[74] Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, and Yap-Peng Tan. Hoi-aware adaptive network for weakly-superviser action segmentation. Proceedings of the Thirty- Secrod International Joint Conference Artificial Intelligence, pages 1722-1730, 2023.
15

---

Supplement to
"Causal Temporal Representation Learning with Nonstationary Sparse Transition"
Appendix organization:
S1 Identifiability Theory
17
ST.1 Proof for Theorem
17
S1.1.1 Identifiability of under optimal g
17
S1.1.2 Identifiability of C under arbitrary
18
$ S1.2 Proof of Corollary
21
$1.3
Proof of Theorem
23
S1.4
Discussion on Assumptions
23
S1.4.1
Mechanism Separability
23
S1.4.2
Mechanism Sparsity
23
$1.4.3
Mechanism Variability
S1.4.4
Mechanism Function Variability
23
$1.4.5
Weakly Diverse Lossy Transition
24
$2
Experiment Settings
24
$2.1
Synthetic Dataset Generation
24
82
$2.2
Real-world Dataset
25
$2.3
Mean Correlation Coefficient
83
Implementation Details
25
S3.1
Prior Likelihood Derivation
25
S3,2
Derivation ELBO
26
S3.3
Reproducibility
27
S3.4
Hyperparameter and Train Details
27
S3.5
Enforce Invertibility
27
S3.6
Encode Domain Variables
27
S4
Extended Related Work
28
S4.1
Causal Discovery with Latent Variables
28
$4.2
Causal Temporal Representation Learning
28
S4.3
Weakly-supervised Action Segmentation
28
16

---

S1 Identifiability Theory
S1.1 Proof for Theorem 1
We divide the complete proof into two principal steps:
1. Firstly, assuming access the optimal mixing function estimation g", demonstrate that under the conditions in our theorem, the estimated clustering result will align with the ground truth up to label swapping. This alignment is due the transition complexity with optimal &z and g' being strictly lower than that with non-optimal úz but still optimal g".
2. Secondly, we generalize the results of the first step to cases where the mixing function estimation g is suboptimal. We establish that for any given clustering assignment, whether optimal or not, a suboptimal mixing function estimation g can not result a lower transition complexity. Thus, the transition complexity in scenarios with non-optimal g will always be at least as high as in the optimal case.
From those two steps, conclude that the global minimum transition complexity can only be attained when the estimation domain variables ût is optimal, hence ensuring that the estimated clustering must match the ground truth up to label swapping. It is important note that this condition alone does not guarantee the identifiability of the mixing function g. Because a setting with optimal iž and non-optimal g may exhibit equivalent transition complexity to the optimal scenario, but does not compromise our proof for the identifiability of domain variables uy Further exploration of the mixing function's 's identifiability g is discussed in Theorem 2 in the subsequent section.
S1.1.1 Identifiability of C under optimal g"
We fist introduce a lemma for this case when we can access an optimal mixing function estimation g*.
X
X
Lemma S1 (Identifiability of C under optimal g'). In addi- tion to the assumptions in Theorem assume that we can also access an optimal estimation of denoted by g", in which the estimated 2t is an invertible, component-wise transformation of a permuted version of 2t. Then the estimated clustering Ć must match the ground truth up to label swapping.
C. A B B
Figure S1: Illustration of ê incor- rectly assigning two different domain subsets of inputs A and B into the same û. The black lines represent the ground truth partition of C and the orange line represent the incorrect do- main partition for set A and B.
Proof. In the first case we deal with optimal estimation g" in which the estimated 2n is invertible, component- -iie transformation of a permuted version of z,, but inaccurate es- timated version of ć, Consider the following example (Figure STT:
With slight abuse of notation, we use C(A) to represent the domain assigned by to all elements in A. and all elements in A have the same assignment. The same argument applies to B.
Then, for an estimated č, if it incorrectly assigns two subsets of input A and B to the same à (Figure ST orange circle), i.e.,
=( but ((A)=C(B)- =B)
(17)
Note that if the ground truth C gives a consistent assignment for and B but estimated č gives diverse assignments, i.e.
( CCB) but C(A) C(B) k;
(18)
it is nothing but further splitting the ground truth assignment in a more fine-grained manner. This scenario does not break the boundaries of the ground truth assignments. Consider two cases in the estimation process:
1. If the number of allowed regimes or domains exceeds that of the ground truth, such more fine-gra graigga assignment is allowed. The ground truth can then be easily recovered by merging domains that share identical Jacobian supports.
17

---

2. If the number of regimes domains matches the ground truth, it can be shown that the inconsistent scenario outlined in Eq. (T7 must occur.
Given that these considerations do not directly affect our approach, they are omitted from further discussion for brevity.
0
0
07
0
1
07
M, =
1
0
0
Mj
=
0 0
M =
0
1
0
1
0
0
0
1
0
0
0
0
0
0
1
0
Figure $2: Comparison of matrices Mi. My, and Me. The elements in red highlight the differences between them.
Then considering the case in Eq. 17 the estimated transition must cover the functions from both A and B, then the learned transition r. must have Jacobian Jm, with support matrix M = MM;+ which the binary addition of M, and M,. such that for all indices M, if any of these two is 1, then the corresponding position in Mr must be 1. That is because if that is not the case, for example, the (a, b)-th location for M,, MM,. and Mr are 1, O, and O. Then
easily find an input region for the (a, b)-th location such that small perturbation can lead to changes in m, but not in mj nor rig, which makes mh unable to fit all of the transitions in A U B which cause contradiction. See the three matrices in Figure S2 for an illustrative example.
By assumption iii, since all those support matrix differ at least spot, which means the estimated version not smaller than the ground truth.
MA
[Mpl and [Mall 2 [Mi).
(19)
and the equality cannot hold true at the same time.
Then from Assumption ii), the expected estimated transition complexity can be expressed as:
Ep Mal = J pD (x(xx11,x)) MM dxx--1 dx..
(20)
Similarly for ground truth one:
Ep|Mul= S Jxxx Po(x
[Mc(x- -.)x,,,,)))MMMxxx.....
dx4-11 dx:.
(21)
Let us focus on the integral of the region A U B, the subset of X x X' mentioned above. If for some area pp(x;:1,,x) x;) = O, then the clustering under this area ill defined since there is no support from data. Hence we only need to deal with supported area. For area that 1x x,) > 0 and from Eq. (19 the equality cannot hold true at the same time, then the estimated version of the integral is strictly larger than the ground-trute truth version for any inconsistent clustering as indicated in Eq. (I7).
For the rest of regions in X' X, any incorrect cluster assignment will further increase the M with same reason as discussed above, then the estimated complexity is strictly larger than the ground truth complexity:
Ep Mal Ep|Maal-
(22)
But assumption ii requires that the estimated complexity be less than or equal to the ground truth. Contradiction! Hence, the estimated Ć must match the ground truth up to label swapping.
S1.1.2 Identifiability of C under arbitrary g
Now we can leverage the conclusion Lemma S1 to show the identifiability of domain variables under arbitrary mixing function estimation.
Theorem S1 (Identifiability of Domain Variables). Suppose that the data 'D are generated from the nonstationary data generation process as described in Eqs. T and ② Suppose the transitions are weakly diverse lossy (Def. 6) and the following assumptions hold:
18

---

i. (Mechanism Separability) There exists ground truth mapping C: X' x X - U determined the real domain indices, i.e., 44 x(x
i. (Mechanism Sparsity) The estimated transition complexity on dataset DD is less than or equal to ground truth transition complexity, i.e., Ep Mal - Ep|Mul
ii. (Mechanism Variability) Mechanisms are sufficiently different. For all u # u', My # My
i.e. there exists index (i.j) such that [Mules * [Murles
Then the domain variables un identifiable up to label swapping (Def
Proof. To demonstrate the complete identifiability of C, independent of the estimation quality of g. we must show that for any arbitrary estimation + o(C), the induced M, for inaccurate estimation g has a transition complexity at least as high as in the optimal g case. If this holds, from Lemma we can conclude that the transition complexity of optimal C = oCC) and optimal g" is strictly smaller than any non-optimal è and any g.
Suppose the estimated decoder and corresponding latent variables g and ê,, respectively, then the following relation holds:
g" (zz) g(2)).
(23)
Since g is invertible, by composing g on both sides, we obtain:
g (z.) g g(z)) h h= g og"
(24)
Let
(25)
we then have:
h 2,
(26)
We aim to demonstrate that under this transformation, if h is not a permutation and component wise transformation, the introduced transition complexity among estimated 2 will not be smaller than the optimal g'
Proposition S1. Suppose [M < [M* then for any permutation a mapping the indices of the
dimensions from M to M", there must exist an index pair (i.J) such that M.. 0 and M Ma( 1.
Proof. An intuitive explanation for this proposition involves the construction of a directed graph G 5 = " where V (1,,,...., and E M. (ii.)) Mi, = 1}. A similar construction can be made for Gzz- It is straightforward that [M = |B which represents the number of edges. Consequently, |M| < <|M' implies that G 5- has more edges than G Since there is pre-defined ordering information for the nodes in these two graphs, if we wish to compare their edges, we need to first establish a mapping. However, if Ex < |Ex- I. no matter how the mapping a is constructed, there must an index pair (i, such that (i.j) É F.g but (o()), a()) E ÆM Otherwise, if such an index pair does not exist, the total number of edges in G ja would necessarily be greater than or equal to that in G MM. contradicting the premise that [M| < [M*)
Lemma $2 (Non-decreasing Complexity). Suppose transitions are weakly diverse lossy as defined in Def. and an invertible transformation h maps the optimal estimation i to the estimated 2, and is neither permutation nor component-wis transformation. Then, the transition complexity on the estimated 2, is not lower than that on the optimal 2;, ie.,
| MM
we provide an illustrative example. Note that the mapping from ground truth z, to optimal estimation 2; is a permutation and element- wise transformation, it does not include mixing, and hence e exists if and only if é exists. Therefore, [M cannot be true.
Proof. The entire proof is based on contradiction. In Figure S3
[M*|= MM The core of the proof requires us demonstrate that [
19

---

++
2
2
2
62
2
(a) Transition graph for ground truth a, and opti- mal estimation 2;
(b) Transition graph for ground truth and arbi- trary estimation
Figure S3: A partial observation of the transition graph among ground truth optimal estimation 27 and arbitrary estimation 2,. For brevity, the index permutation is assumed to identity, i.e., (t) i.
Suppose the transitions are weakly diverse lossy as defined in Def. [ then for each edge žt,i - 1e+1, in the transition graph, there must be a region of žr,i such that only 21+1, influenced by 2t,i- Consequently, the corresponding Ætti and 1, not independent, since mixing process can cancel the influence of zt.i. Therefore, the edge 2,,; + ătt1 the estimated graph must exist.
Note that without the weakly diverse lossy transition assumption, this argument may not hold. For example, if 241, can be expressed as function that does not depend on 24.5, then even though the edge ž + 21,1, exists, the estimated edge 2,, - 24+, may not exist. This could occur if, after the transformation h. the influences in different paths from 24,4 4 cancel out with each other.
Necessity Example An example that violates the assumption is follows:
1,1 2t,i + 62+1,
1t 21,1 + f1
& 2;
2 24 2j
Here, the mapping from z is invertible, Writing down the mapping from à; to 2c+1, particularly for 2411f, yields;
E411 = (z,, E441,5) - (Ze,i + Ee+115)
6t41, - 6x41.
Clearly, this is independent of â,, Hence, this scenario, the edge the estimated graph does exist. This explains the necessity for the weakly diverse lossy transition assumption. Furthermore, it can be seen that violating the weakly diverse lossy transition assumption would require a very specific design, such as the transition in an additive noise case and the transition on & being linear, which is usually not the case in real-world scenarios. Generally, this requires that the influences from different paths from Z to &4t1, cancel each other out, a condition that is very challenging to fulfill in practical settings.
Permutation Indexing One may also ask about the permutation of the index between and 2, Since the transformation is invertible, the determinant the Jacobian should be nonzero, implying the existence of a permutation such that
(i, (i)) supp(Jn), Vi E [m]]
Otherwise, if there exists an such that [Jn],, 0 or [Jn).: = O, such a transformation cannot be invertible. We can utilize this permutation to pair the dimensions z, and é,.
Since each ground-truth edge is preserved in the estimated graph, by Proposition SI the inequality | |M cannot hold true. Thus, the lemma is proved.
20

---

Then, according to this lemma, the transition complexity [Ma of the learned m, should be greater than equal [Ma which is the complexity when using an accurate estimation of g'. This relationship can be expressed as follows:
Mal My
By Lemma S1 the expected complexity of the estimated model Ep Ep|Ma is strictly larger than that of the ground truth Ep Mul. This implies the following inequality:
Ep|Mal 2 Ep|Mal Ep|Mal.
(27)
However, Assumption iii requires that the estimated complexity must be less than equal to the ground-truth complexity, leading to a contradiction. This contradiction implies that the estimated must match the ground truth up to label swapping. Consequently, this supports the conclusion of Theorem
S1.2 Proof of Corollary
Corollary S1 (Identifiability under Function Variability). Suppose the data DD is generated from the nonstationary data generation process described in (1) and (2). Assume the transitions are weakly diverse lossy (Defi 6). and the mechanism separability assumption aloong with the following assumptions hold:
V. (Mechanism Function Variability) Mechanism Functions are sufficiently different. There exists K E N such that for all u + u', there exists k < K, M # M i.e. there exists index (i,j) such that [M 8.3 + [M
vi. (Higher Order Mechanism Sparsity) The estimated transition complexity on dataset D no more than ground truth transition complexity,
Ep
LIMM
< ED
M
(28)
Then the domain variables 4n identifiable up to label swapping (Def 2
Proof. With a strategy similar to the proof of Theorem we aim to demonstrate that using an incorrect cluster assignment Ć will result in Σ k= [M being higher than the ground truth, thereby still enforcing the correct U4.
Differing slightly from the approach in Theorem 1 this setting, we will first demonstrate that under any arbitrary assignment, the estimated complexity is no lower than the complexity in the ground truth, i.e., Σ [M > >>_11 [M*].
First, we address the scenario where diffferent domains have the same transition graph but with different functions, as otherwise, the previous lemma [ST still applies. In cases where the same transition causal graph exists but the functions differ, assumption y indicates that there exists an integer & such that M o M meaning the ground truth support matrices are different. However, due to incorrect clustering, the learned transition must cover both cases. To substantiate this claim, we need to first introduce an extension of the -on-decrgaiing complexity lemma.
Lemma S3 den-eecceas Complexity under Mechanism Function Variability). Suppose there exists an invertible transformation h which maps the ground truth z, to the estimated 2., and it is neither a permutation nor a component-wie wise transformation. Then, the transition complexity on the estimated 2, is not lower than that on the ground truth zı. iee.
K
Σ
M
1=1
21

---

Proof. We can extend the notation of the edges e to the higher-order case ex to represent the existence of a non-zero value for the k-th order partial derivative ø'm., Under the weakly diverse lossy 225
transition assumption, it is always possible to find a region where the influence in ex cannot be component-wise
canceled in ě². In this region, the mapping from 2i to & can be treated transformation, since the influence of z other than z is zero due to the lossy transition assumption. It is important to note that there is also indexing permutation issue between 2, and ăy; the same argument in the permutation indexing part of the proof of lemma S3 applies.
Since M represents the support of the k-th order partial derivative, this implies that [M*] 1 implies [M
for all k' I k. We aim to show that if for the transition behind edge Z1, + 2,+1,i, there exists a K such that [Milk are different for two domains, then one of them must be polynomial with order K 1. For this domain, [M when k 1,2..... K
and [M when k 2 K.
To demonstrate that the non-decreasing complexity holds, we need to show that after an invertible transformation h to obtain the estimated version. [Mulk, cannot be zero for k < K - 1, which can be shown with the following proposition.
Proposition S2. Suppose f is a polynomial of order ls with respect to x. Then, for any invertible smooth function h, the transformed function f = h- of order K', when K' < k.
1--1 a f o h cannot be expressed by polynomial
Proof. Let's prove it by contradiction. Suppose f == h- f o h can be expressed as a polynomial of order K' < k. It follows that the function f(xx = C has k' roots (repeated roots are allowed), since h is invertible. Therefore, h o f(x) h(C)) also has the same number of k' roots. By definition, f = f h, which means foh h(C) has k roots. However, since h is invertible, or equivalently it is monotonic, the equation f o h = hC()) having k' roots implies that f(x = C' has roots k'. Yet, since is a polynomial of order k, it must have l roots, contradicting the fundamental theorem of algebra, which means that they cannot have the same number of roots. Hence, the proposition holds.
The advantage of support matrix analysis is that, provided there exists at least one region where the support matrix is non-zero. the global version on the entire space will also be non-zero. Based on the definition of diverse lossy transition in Def. 6 it is always possible to identify such a region where for an edge 2t,; + 2++1.j. the mapping from 2t+1,, to 2t-1, can be treated as a component-wise relationship. This is because no other variables besides 2t+1.j change in conjunction with 2,,; to cancel the effect. Therefore, proposition (S2 applies, and as a result, the complexity is nondecreasing. Thus, the lemma is proved.
With this lemma, we have shown that for an arbitrary incorrect domain partition result, the induced ground-truth transition complexity is preserved after the invertible transformation h. This partition effectively combines two regions, as illustrated in Figures ST and S4 Consequently, the transition complexity has the following relationship:
M = MA + M
Here, u and u' represent the ground-truth values of the domain variables, and denotes the estimated version, defined as the binary addition of the two ground truths.
By assumption v the two ground truth transitions' complexity M MM are different, then with the same arguments in the proof of the lemma we can show that the expected transition complexity with wrong domain assignment over the whole dataset is strictly larger than the ground truth com- plexity with correct domain assignment. And it is easy to see that when the estimated latent variables are equal to the ground truth, 2, 2, then the lower bound is achieved when the estimated domains are accurate. Note that this argument is not sufficient condition to say that the estimated 2, is exactly the ground truth 2, or an optimal estimation of it, since there can be other formats of mapping from 2, to 2, that generate the same complexity. But this is sufficient to prove that by pushing the complexity to small, the domain variables 24 must be recovered up to label swapping. This concludes the proof.
22

---

$1.3 Proof of Theorem 2
Theorem S2 (Identifiability of the Latent Causal Processes). Suppose that the data is generated from the nonstationary data generation process ① 2}. which satisfies the conditions in Theorem 7 and Lemmal 7 then the domain variables un are identifibble up to label swapping (Def. 2 and latent variables are identifiable up permutation and component-wis transformation (Def.
Proof. From Theorem T the domain variables u, are identifiable up to label swapping, and then use the estimated domain variables in Lemma[] [ the latent causal processes are also identifiable, that is, Zt are identifiable up to permutation and a component-wis transformation.
$1.4
Discussion on Assumptions
$1.4.1 Mechanism Separability
Note that we assume that there exists a ground truth mapping C X' X' - gives a domain index based on Xt. The existence of such mapping means that the human can tell what the domain based on two consecutive observations. If two observations are not sufficient, then it can be modified to have more observation steps as input, for example X<x even full sequence X[1:7]- If the input has future observation, which means that X>+ included, then this only valid for sequence understanding tasks in which the entire sequence will be visible to the model when analyzing the time step For prediction tasks or generation tasks, further assumptions on C such as the input only contains x<t should be made, which will be another story. Those variants are based on specific application scenarios and not directly affect our theory, for brevity, let us assume the two-step case.
$1.4.2 Mechanism Sparsity
This is a rather intuitive assumption in which we introduce some form sparsity the transitions, and our task is to ensure that the estimated transition maintains this sparsity pattern. This requirement enforced by asserting an equal or lower transition complexity as defined in Assumption [i] Similar approaches, grounded in the same intuition, are also explored in the reinforcement learning setting, as discussed in works by Lachapelle et al. [18] and Hwang et al. [45). The former emphasizes the identifiability result of the independent components, which necessitates additional assumptions. In contrast, the latter focuses on the RL scenario, requiring the direct observation
the latent variables involved in the dynamics, which leaves significant challenges in real- world sequence- understanding tasks, where the states are latent. Some studies have also explored the application of sparsity constraints the estimated latent representations [46]. And it is also extensively discussed in which such a sparsity constraint was added to the mixing
in the nonlinear ICA literature 33 19 function.
$1.4.3 Mechanism Variability
The assumption of mechanism variability requires that causal dynamics differ between domains, which requires at least one discrete edge variation within the causal transition graphs. This assumption typically considered reasonable in practical contexts; humans identify distinctions between domains only when the differences are substantial, which often involves the introduction of a new mechanism or the elimination of existing one. Specifically, this assumption requires minimal alteration, single edge change in the causal graph, to be considered satisfied. Consequently, as long as there are significant differences in the causal dynamics among domains, this criterion is fulfilled.
$1.4.4 Mechanism Function Variability
In this section, we will further discuss the mechanism function variability introduced in Corollary One might question the necessity of this assumption. To illustrate this issue, we claim that if we only assume that the mechanism functions differ across domains but without this extended version of the variability assumption, i.e., for u * u', m, + m,,, then under this proposed framework, the domain variables u, generally unidentifiable.
In Figure S44 we present a simple example of the space of z given fixed 2,, For the sake of brevity, assume that there are two domains. By the mechanism separability assumption i the space Z
23

---

of Z1+1 is divided into two distinct parts, each corresponding to one domain. In this illustration, C denotes the partition created by the ground truth transition function:
m (Zc,E)
if u1+1 L,
Z++1
(29)
m2 (Z),6)
if 11411 = 2.
C
Then the question arises: when the domain assignment incorrect, that is, č * C, can we still get the same observational distributions, equivalently, can we obtain the same distribution for Zy+1?
Z
A
B
The answer is yes. For the ground truth transition, mi (2), E A C and m2(24, <) E B U D. In the case of an incorrect partition C. it is sufficient to have mn (z,, c) E AUB and my <) C.D. Ensuring that the conditional distribution p z,) is matched every- where,
C
C
0 D
create two different partitions domains, yet still obtain exactly the same observations. That makes the domain variables u, unidentifiable in the general case.
Figure S4: Correct domain separation C and incorrect domain separation Ć of Z given fixed z₁.
How does the previous mechanism variability assumption work? the assumption of mecha- nism variability (Assumption inn. the support matrices of the Jacobian of transitions across different domains differ. Consider scenario where the ground truth partition C, denoted by A, C' B, D.
an incorrect estimation occurs, where our estimated partition is Ć, represented as A, B C, then the estimated transition in domain one should cover the transitions in both A and B, and similarly for the second domain. This leads to an increase in complexity within the estimated Jacobian support matrix, as discussed in the previous sections. Consequently, this complexity forces the sets and C to be empty, resulting in ć converging C.
How about mechanism function variability? Roughly speaking, and as demonstrated in our experiments, the mechanism variability assumption previously discussed is already sufficient to identify domain changes in both synthetic and real-world settings. This sufficiency arises because the assumption only requires a single differing spot, even though some transition functions behind some edges may persist across different domains. As long as there is one edge spot that can separate the two domains, this condition is met. In the relatively rare case where all edges in the causal dynamic transition graphs are identical across two different domains and only the underlying functions differ, we can still demonstrate identifiability in this scenario by examining differences in the support the higher-order partial derivative matrices.
S1.4.5 Weakly Diverse Lossy Transition
The weakly diverse lossy transition assumption requires that each variable the latent space can potentially influence set of subsequent latent variables, and such transformations are typically non-invertible. This implies that given the value of Zt+1, recover the previous Z,; equivalently, this mapping is not injective. Although this assumption requires some explanation, is actually considered mild in practice. Often real-world scenarios, different current states may lead to identical future states, indicating a loss of information. The "weakly diverse" of this assumption suggests that the way information lost varies between different dimensions, but there is some common part among them, hence the term "weakly diverse". In the visualization example shown in Figure 4 we can clearly see this pattern, in which the scene is relatively simple and it is very likely that in two different frames, the configuration of the scene or the value of the latent variables are the same but their previous states are completely different.
generally challenging to precisely
S2
Experiment Settings
$2.1
Synthetic Dataset Generation
The synthetic dataset is constructed in accordance with the conditions outlined in Theorems I [nd Transition and mixing functions synthesized using multilayer perceptrons (MLPs) initialized with
24

---

random weights. The mixing functions incorporate the LeakyReLU activation function to ensure invertibility. The dataset features five distinct values for the domain variables, with both the hidden variables and the observed variables x, set to eight dimensions. A total of 1. 1,000 sequences of domain variables were generated. These sequences exhibit high nonstationarity across domains, which cannot be captured with a single Markov chain. This was achieved by initially generating two distinct Markov chains to generate two sequences of domain indices. Subsequently, these sequences were concatenated, along with another sequence sampled from discrete uniform distribution over the set {1, 2, 3, 4, 5). representing the domain indices.
For each sequence of domain variables, we sampled batch size of 32 sequences of hidden variables z, begin- ning from a randomly initialized initial state Zo- These sequences were generated using the randomly initialized multilayer perceptron (MLP) to model the transitions. Observations X were subsequently generated from 2; using the mixing function specified in Eq. [ Both the transition functions in the hidden space and the mix- ing functions were shared across the entire dataset. A summary of the statistics for this synthetic dataset is pro-
Table S1: Synthetic Dataset Statistics
Property
Value
Number of State
Dimension of z₁
Dimension of
8
Number of Samples 32,000
Sequence Length 15
vided in Table [S]] For detailed implementation of this data generation process, please refer to accompanying code in Sec. S3.3
S2.2 Real-world Dataset
Hollywood Extended frames containing sequences of 16 different daily actions such walking sitting from 69 Hol- lywood movies. On average, each video comprises 5.9 segments, and 60.9% of the frames are background.
The Hollywood 38 dataset contains 937 video clips with a total of 787,720
CrossTask The CrossTask 339 dataset features videos from 18 primary tasks. According to [40]. we use the selected 14 cooking-related tasks, including 2552 videos with 80 action categories. On average, each video in this subset has 14.4 segments, with 74.8% of the frames classified as background.
S2.3 Mean Correlation Coefficient
MCC, standard metric in the ICA literature, is utilized to evaluate the recovery of latent factors. This method initially computes the absolute values of the correlation coefficients between each ground truth factor and every estimated latent variable. Depending on the presence of component- wise invertible nonlinearities in the recovered factors, either Pearson's 's correlation coefficients or Spearman's 's rank correlation coefficients are employed. The optimal permutation of the factors is determined by solving linear sum assignment problem on the resultant correlation matrix, which executed in polynomial time.
S3
Implementation Details
$3.1 Prior Likelihood Derivation
Let us start with illustrative example of stationary latent causal processes consisting of two time- delayed latent variables, i.e., 2, = [21,e, 22,,], i.e., 2i, = m, m,(2,-1, G. with mutually independent noises, where omit the u, since it is just index to select the transition function m, Let write this latent process as a transformation map m (note that we overload the notation m for transition functions and for the transformation map):
21,-1
21,111
22
22
(30)
21,t
61,t
22,t
62,t
25

---

By applying the change of variables formula to the map m. we can evaluate the joint distribution of the latent variables p1 22, 22,t) as:
22 -
(31)
where Jm is the Jacobian matrix of the map m, which is naturally a low-triangular matrix:
0
J m
0zi,t
D21.1
Basut del.
0
021,t-1
d=2,
Ozg,t
052. Dzg,1
0
881.e-1
822.c-1
8ea.c
Given that this Jacobian is triangular, we can efficiently compute its determinant thermore, because the noise terms are mutually independent, and hence G, 1 Cg. for j + and 1 Zt-1, write the RHS of Eq. 31 as:
II; deid Fur- Jer."
p(21,111 x ple,, (2,)) det Jml p(21,-1, 22,1-1 22,)
(because e Z
=p(21.-11 221-1 Il p(e..)/ /et Jml Jml
(because E1,t 1 E2,c)
(32)
Finally, by canceling out the marginals of the lagged latent variables p(21,1 22, on both sides, we can evaluate the transition prior likelihood as:
p(z 22 1211111,22.0 22,--1 = II p(e,t)/ det Jml = II p(e,,) x |det
(33)
Now we generalize this example and derive the prior likelihood below.
Let {m, 1,233 be set of learned inverse transition functions that take the estimated latent causal variables, and output the noise terms, i.e., J. r, (u, 2 2-
Design transformation A B with low-triangular Jacobian follows:
In
2 11-1,22) 2c]
mapped [ with JA-DB
8m
(34)
diag
däye
Similar to Eg
33 we can obtain the joint distribution of the estimated dynamics subspace as:
log.p( A) log (Z.
p(e,() + log (Jdet (JA-BB())
(35)
3=1
Mutually independent noise
log (2, 21-1, Σ log ui) + Σ log
dm 05,,
(36)
1=1
S3.2 Derivation of ELBO
Then the second part is to maximize the Evidence Lower BOund (ELBO) for the VAE framework, which can be written
26

---

ELBO
log Pauta (f (fx+)i=)) - DxL(Io((2))6 (xttim)) || Pamal(z))-n (xx
Ex, log Pauna ( ((x )/-1 {z) [m - D (go fxi (m)) |l paia Zz Ji=1 fxi {x1}(=n)
Ez, log Phun X | {ze |i= - Ex: (log qoi 12 (xx)==)) log pana ( (t |
Exe
Pauaa
{2e )f=1) + log Puana ( (z )E=1) ) log gu ( | {xx (x))(=1)
T
Exn
log pram (Ne, 21) +
log paina (Z: 11-1, Us)
log qo (z xi)
ERocon
CRLD
(37)
S3.3 Reproducibility
All experiments are performed on a GPU server with 128 CPU cores, 1TB memory, and one NVIDIA L.40 GPU. Our code is also available via htttp:://githii com/ wlaagnchhnnsnng/crrmmm For synthetic experiments. we run the baseline methods with implementation from https://gtthbbbb http:/tttuu..oo//eieieeee cww/weirayao/ /Leap and https For real-world experiments, the implementation based on https /tthhhhhcccc///eeeeeaaaaaarooooo
//github.com/isee wwworatrry/corr24444444
S3.4 Hyperparameter and Train Details
For synthetic experiments, the models were implemented in PyTorch 2.2.2. We trained the VAE network using the AdamW optimizer with a learning rate of 5 10 and a mini-batch size of 64. Each experiment was conducted using three different random seeds and reported the mean performance along with the standard deviation averaged across these seeds. The coefficient for the L2 penalty term was set to 10 which yielded satisfactory performance in our experiments.
the most accurate way to enforce sparsity through the Lo norm. However, since calculating the gradient for it is challenging, Lp norms are commonly used as approximations. We also tested alternative settings such as Li penalty larger coefficients for La norm, and we found that the setting used in this paper (L2 with coefficient 1 10 10-1) provided the best stability and performance.
All other hyperparameters of the baseline methods follow their default values from their original implementation. For real-world experiments, we follow the same hyperparameter setting from the baseline ATBA method. In the Hollywood dataset, we used the default 10-fold dataset split setting and calculated the mean and standard derivation from those 10 runs. For the CrossTask dataset, we calculate the mean and standard derivation using five different random seeds.
S3.5 Enforce Invertibility
In our experimental setup, using reconstruction loss already provides strong identifiability results (MCC>0.9) (5)), and it is widely used in the identifiability literature. During our experiments, we found that flow-based methods are usually less efficient and typically take longer to converge. Since our main contribution is to address the challenge of unknown domain variables, this choice is orthogonal our thcoretical contribution. Therefore, we followed existing work the design of the estimation for mixing function. As also mentioned in [47] the reconstruction loss- based framework can definitely be extended to flow-based methods, especially in environments where invertibility is critical issue and computation is not a top priority in the estimation process.
S3.6 Encode Domain Variables
As introduced in 4 utilize U distinct networks to capture the different transitions. One may question whether this choice introduces redundant parameters and suggest employing parameter-sharing networks, as in (2 45 However, our decision to use separate transition networks is based on the assumptions regarding the complexity of the transition functions, which we regularize through the sparsity of the Jacobian matrix. Implementing parameter-sharing techniques would significantly complicate the optimization process, as updating parameters for one domain would immediately alter the Jacobian matrix for another domain.
Despite using individual transition networks, they remain lightweight compared to entire framework. Even in synthetic scenarios where the framework is relatively small, each transition network accounts for only approximately 2.3% % of the total parameters. This proportion becomes even smaller in weal-world applications with larger decoder frameworks.
27

---

S4
Extended Related Work
$4.1 Causal Discovery with Latent Variables
Various studies have focused on uncovering causally related latent variables. For example, 53. use vanishing Tetrad conditions [5]] or rank constraints to detect latent variables in Gansai-Gaussaan models, whereas [52 rely on non-Gaussianity in their analyses of linear, non-Gaussian models. Additionally, some methods seek to identify structures beyond latent variables, leading to hicrarchical structures. Certain hicrarchical model-based approaches assume tree- like configurations, as seen in [56 [9], while other methods consider more general hierarchical structure [55] 50). Nonetheless, these approaches are restricted to linear frameworks and encounter increasing difficulties with complex datasets, such as videos.
S4.2 Causal Temporal Representation Learning
In the context of sequence or time series data, recent advances in nonlinear Independent Component Analysis (ICA) have leveraged temporal structures and nonstationarities to achieve identifiability. Time-contrastive learning (TCL) [T] exploits variability in variance across data segments under the assumption of independent sources. Permutation- contrastive learning (PCL) [II] discriminates between true and permuted sources using contrastive loss, achieving identifiability under the uniformly dependent assumption. The i- VAE [I3 uses Variational Autoencoders to approximate the joint distribution over observed and nonstationary regimes. Additionally, ()-CITRIS (60 61 utilizes intervention target information to identify latent causal factors. Other approaches such as LEAP 22 and TDRL. 23 leverage nonstationaritic from noise and transitions to establish identifiability. GCIM 62 builds on the theoretical insights from LEAP by implementing a clustering algorithm address the challenge of unobserved domain indices. However. it does not achieve identifiability of the domain variables. CaRiNG 63] extended TDRL to handle non-invertible generation processes by assuming sequence-wist wise recoverability of the latent variables from observations.
All the aforementioned methods either assume stationary fixed temporal causal relations or that the domain variables controlling the nonstationary transitions are observed. To address unknown or unobserved domain variables, HMNLICA [15] integrates nonlincar ICA with hidden Markov model to automatically model nonstationarity. However, this method does account for the autoregressive latent transitions between latent variables over time. IDEA 26 combines HMNLICA and TDRL by categorizing the latent factors into domain- variant and domain-invariant groups. For the variant variables, IDEA adopts the same Markov chain model as HMNLICA, while for the invariant variables, it reduces the model to a stationary case handled by TDRL. Both iMSM 24 and NCTRL 25 extend this Markov structure approach by incorporating transitions in the latent space but continue to assume that the domain variables follow a Markov chain.
Moreover, causal temporal representation learning has been explored the reinforcement learning literature [64] These works primarily focus on the relationship hetween states and actions, typically relying on direct state observations. In contrast, our setting involves recovering meaningful latent causal variables from observational data.
45..
S4.3 Weakly-supervised Action Segmentation
Weakly-supervised action segmentation techniques focus on dividing a video into distinct action segments
using training videos annotated solely by transcripts [38 65
43
66
42 67
74). Although these
methods have varying optimization objectives, many employ secudo-segmentatioon for training by aligning video sequenees with transcripts through techniques like Connectionist Temporal Classification (CTC) (65]. Viterbi [1 43 42
70 73, or Dynamic Time Warping (DTW) (67 (69 For instance, (65] extends CTC to consider visual similarities between frames while evaluating valid alignments between videos and transcripts. Drawing inspiration from speech recognition, [71] 41 73 utilize the Hidden Markov Model (HMM) to link videos and actions. 66) initially produces uniform segmentations and iteratively refines boundaries by inserting repeated actions into the transcript.
4% introduces an alignment objective based on explicit context and length models, solvable via Viterbi, to generate pseudo labels for training a frame- wise classifier. Similarly, 44 and
770 propose novel learning objectives but still rely on Viterbi for optimal pseudo segmentation. Both 67 69
DTW align videos to both ground- totut and negative transcripts, emphasizing the contrast between them. However, except for (66) these methods require frame-hy-frame calculations, making them inefficient. More recently, alignment-free methods have been introduced to enhance efficiency. (8) learns from the mutual consistency between frame-wise classification and category/length pairs of a segmentation.
[44 enforces the
output order of actions to match the transcript order using novel loss function. Although POC set-supervised. it can be extended transcript supervision, making its results relevant for comparison.
44 is primarily
28

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We state our contributions in the abstract and explicitly highlight them at the end of the introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
A dedicated limitation discussion section is provided in the Appendix 7
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- specifleciiii..nnn asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions
online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification: All assumptions together with necessary definitions are provided in the main text, proof sketch is also provided in the main text, and the complete proof is provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and crfseeeeferenced.
29

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code to reproduce the experiment and the figures are provided as mentioned in Appendix S3.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The code and the data to reproduce the experiment and the figures are provided as mentioned in Appendix S3.3
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
http://iippsccccpullttp
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
30

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://niaa.cc/ppin for more details.
Buidee//DodeSuumiissioooPPo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Detais
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The experiment settings both synthetic and real real-world experiment are discussed in setup parts in Sec. [ The Appendix. aSso Appendix,|S3.4aaso discussed the training details. Those information can also be easily obtained from the released code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
The synthetic experiments were supported by mean values and standard derivation from the mean, with analysis on the stability the training. For real-world setting, we report error bars for our model. We run the experiments following existing setting on the data splitting or random seeds to calculate the mean and std. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether error the standard deviation or standard error of mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions. the authors should be careful not to show in tables or figures symmetric emror bars that would yield results that are of range (e.g. negative
rates). If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
31

---

Justification: See Appendix S3.3 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:////aarrr..ccco///aa//aaaaa
Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics.
Guidelincs:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification:
The broader impact is discussed in Appendix
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose such risks.
Guidelines:
32

---

The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model. for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code. data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The codebase together with the datasets used our paper are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL.
name of license (e.g.. . CC-BY 4.0) should be included each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details about the code together with the synthetic dataset can be found in anonymized url provided in Appendix $3.3
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
33

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, ble), such as institution conducting the review.
include any information that would break anonymity applica-
34


Reviews:
[{"review_id": "bl5LZku99a", "reviewer": "Reviewer_3MKp", "summary": "The paper introduces CtrlNS, a causal temporal representation learning framework based on a sparse transition assumption to identify distribution shifts without strong prior knowledge of the domain variables. Theoretical and experimental results show CtrlNS effectively identifies distribution shifts and latent factors, outperforming existing methods.", "questions": "- It seems that the framework requires the prior knowledge of the number of environments $U$. This assumption should be made more explicit. Also, what happens under the misspecification of $U$?\n- For the mathematical rigor, the statements need to take measure-zero sets into account. For example, I think the conditions $p(S_{t, i, j})>0$ (line 136), $p(S_{t, i})>0$ and $p(S_{t, i, j}\\setminus S_{t, i})>0$ (line 141) should be included, just to name a few.\n- The framework requires $U$ different transition networks, which could be a potential weakness, especially in terms of scalability. The discussion on this point would further strengthen the paper. For example, an efficient parameter sharing of networks [3] may alleviate such issues.\n- The proposed work assumes no contemporaneous causal relationships. (I know this is a common assumption in this line of work) How can this assumption be relaxed? With the interventional data, is it possible for the proposed framework to generalize to the existence of instantaneous effects, maybe similar to iCITIRS [1]?\n- A number of works in RL setting also consider the identifiability under the non-stationary process [2, 3]. For example, [3] considers the identifiability of both domain variables and causal structures. I suggest the authors include these related works in different areas.\n- The utilization of mechanism sparsity for identifiability is also explored in [4]. How is it different from the sparse transition assumption in CtrlNS?\n\n(minor)\n\n- Style file seems to be different from the official NeurIPS template (e.g., fonts).\n\n***References***\n\n[1] Lippe, Phillip, et al. \"Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems.\" *The Eleventh International Conference on Learning Representations*.\n\n[2] Feng, Fan, et al. \"Factored adaptation for non-stationary reinforcement learning.\" *Advances in Neural Information Processing Systems* 35 (2022): 31957-31971.\n\n[3] Hwang, Inwoo, et al. \"Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning.\" *International Conference on Machine Learning*. PMLR, 2024.\n\n[4] Xu, Danru, et al. \"A Sparsity Principle for Partially Observable Causal Representation Learning.\" *International Conference on Machine Learning*. PMLR, 2024.", "limitations": ".", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Causal temporal representation learning under nonstationarity is an important problem. The paper replaces somewhat unrealistic and strong assumptions on the domain variable made by the prior works.\n- Theoretical analysis provides a rigorous foundation for the proposed framework. While I did not check the proofs in detail, the proof flows are convincing, and the results are intuitive and reasonable.\n- Another strength of the paper is its evaluation on realistic task setup (i.e., weakly supervised action segmentation). This illustrates the practical applicability of the proposed method.\n- Finally, the paper is well-written and well-organized.", "weaknesses": "The authors argue that the paper derives identifiability results *“without prior knowledge of the domain variables”* (e.g., line 46, 61). However, this is an overstatement since CtrlNS still requires some prior knowledge or assumptions, such as the ground-truth number of different domains $U$. Moreover, while I understand it does not rely on Markov assumption of prior works (e.g., NCTRL), the proposed framework still requires a different set of assumptions such as weakly diverse lossy transition and mechanism sparsity. In fact, I am not sure whether the authors’ assumption can be claimed as “weaker” compared to prior works. Therefore, I feel those statements need to be toned down, and more discussion on the comparisons of those assumptions is suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our work and contribution, and we also thank the reviewer for providing valuable questions and suggestions. Please kindly find our response below:\n\n> W1+Q1. Claim on Prior Knowledge\n\nWe appreciate the reviewer for their valuable suggestions to make our presentation more precise. We have revised this claim to a more moderate statement: \"doesn't rely on knowledge of the prior distribution of the domain variables.\" We have explicitly included the requirement of a predefined number of $U$ in the updated paper.\n\n> W2. Assumption Comparison with NCTRL\n\nWe completely agree that there is no strict winner or loser between our assumptions and those of NCTRL. We have weaker assumptions on the distribution of domain variables but place additional assumptions on transition functions. We intend for our method to serve as an alternative solution when the distribution of domain variables is unknown.\n\n> Q1. What happens under misspecification of $U$?\n\nSpecifying an insufficient number will definitely hurt the identifiability, causing multiple domains to become entangled. However, as discussed in Appendix S1.1.1 (lines 709 to 711), as long as the allocated number of $U$ is greater than or equal to the ground truth, identifiability can still be achieved. This also suggests a potential future work direction on how to further relax this requirement and automatically find the number of $U$, starting from a sufficiently large number and gradually decreasing to a suitable value.\n\n> Q2+Q(minor). More Precise Mathematical Statements and Style File Issue\n\nWe thank the reviewer for pointing this out. We have added those conditions and carefully proofread our statements to ensure their precision when updating the draft. Regarding the style file issue, we have identified that the problem stems from the `lmodern` package, which we have removed in the updated version.\n\n> Q3. Why Separate Transition Networks\n\nWe thank the reviewer for raising this question and completely agree with the reviewer's concern. The reason we use different transition networks is due to our assumptions on the complexity of the transition functions, where we regularize using the sparsity of the Jacobian matrix. Using parameter-sharing tricks would make the optimization problem very difficult, since in this setting, updating the parameter for one domain will immediately change the Jacobian matrix for another domain. Despite using separate transition networks, they are lightweight compared with the whole framework; even in a synthetic setting where the whole framework is relatively small, each transition network only takes ~2.3% of the parameters. This percentage is even lower in real-world cases with larger encoder-decoder framework. We have included a discussion of parameter-sharing networks in our updated version.\n\n> Q4. Contemporaneous Relation\n\nWe noticed that there is concurrent work [5] that also takes advantage of the sparsity of the transition to establish identifiability with instantaneous relations. Their setting is stationary, but we believe our work can be utilized to further extend [5] to a nonstationary setting.\n\n> Q4. Interventional Data\n\nWe thank the reviewer for such a valuable suggestion. We believe that leveraging interventional data can further relax the assumptions made in this work. For example, the one-edge difference among different domains may not be necessary with interventional data, as we can always compose paired data to separate the domains. We leave further discussion in this line as a future work direction.\n\n> Q5. Related Work in RL Community\n\nWe thank the reviewer for reminding us to discuss related works including [2,3] in RL. RL focuses more on the relation between states and actions, while we focus more on finding meaningful representation from observational data. For instance, [3] uses direct observation of states in RL environments, while our setting requires recovering meaningful latent causal variables from observational data. We have included more discussions within this line of research in the updated version.\n\n> Q6. Difference from [4]\n\nThanks for pointing to this paper. We noticed that [4] also utilized sparsity constraints to establish identifiability results. However, the way sparsity constraints are applied is different from our method. The sparsity in [4] was placed on the values of latent variables $\\mathbf{Z}$, as there is no temporal process in their setting. In our case, we place the sparsity constraint on the transition of the latent variables, allowing the values of latent variables in our method to be almost arbitrarily dense.\n\n[5] Li, Zijian, et al. \"On the Identification of Temporally Causal Representation with Instantaneous Dependence.\" 2024, arXiv, https://arxiv.org/abs/2405.15325.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJHLoQLVNy", "reviewer": "Reviewer_Zo2n", "summary": "The paper focuses on temporally causal representation learning, where the goal is to recover a latent causal process from nonstationary observation sequences. Existing works represent the source of the nonstationarity by either known domain variables or (autocorrelated) unknown domain variables with Markov structure. The paper proposes a sparsity based constraint on the latent transitions to achieve the identifiability of the domain variables. Coupled with the identified domain variable, they make use of the sufficient variability assumption to identify the latent causal process. Based on this theoretical framework, they introduce a sequential VAE based model, which encourages the independent noise variables and the sparse transitions. They evaluate (i) the performance on the latent process identifiability on a synthetic dataset following their model definition, and (ii) the usefulness of their model on a weakly-supervised video action segmentation task on real-world datasets of daily actions and cooking.", "questions": "* Some assumptions are motivated in lines 219-224 as being aligned with human intuition: “separability states that if human observers cannot distinguish between two domains, it is unlikely that automated systems can achieve this distinction either. Secondly, variability requires that the transitions across domains are significant enough to be noticeable by humans, implying that there must be at least one altered edge in the causal graph across the domains.” I find it hard to follow this reasoning. Why are these the cases?\n* For both experiments, architectural details are missing. Besides, some information on the synthetic experimental setups is missing: what are the functions $\\mathbf{m}$ and $\\mathbf{g}$ used?, What is the train/val/test split?, What parts of the experiment change with the changing seed?\n* It is not clear to me what the model predicts on the action segmentation task. I think it predicts the action class per frame given the video frames and the action order. If so, why do you report MoF, IoU and IoD, instead of accuracy? How do you provide the weak-supervision information to the model? Why is the baseline NCTRL excluded from this task? Its generative model seems to fit the task well.\n* It is not clear what the sparsity loss is. In line 266, it is said to be the L2 norm of the parameter in the transition estimation functions. Which parameter is this?", "limitations": "* The main limitation, the assumptions being untestable for a real-world setup, is not discussed in the main text.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* Compared to the existing work, the paper tackles the identifiability of the domain variables from a new perspective: the domain variable's effect on the transition functions. This perspective is in line with seeing distribution shifts as interventions to the underlying system, and could be valuable in some related tasks such as learning causal relations by taking actions.\n* I appreciate the effort for the real-world experiment, though it only evaluates the prediction of the domain variable.", "weaknesses": "* As is common in identifiability literature, the assumptions of the theoretical framework do not seem to be testable on a real-world setup.\n* Definition 6 (Weakly diverse lossy transitions) seems unrealistic and is not clear from the text: (i) The motivation for the “Lossy” is not clear to me, how is there a causal link between two variables, while changing the causal parent does not affect the effect variable? (ii) On the clarity of the definition: perhaps, some graphical model examples could help here. In addition, the assumption is not discussed in the main text, except stating in lines 227-228 that it is “a mild and realistic condition in the real-world scenarios, allowing for identical future latent states with differing past states”. Why is it mild and realistic?\n* It is not clear to me why having domain variables without a Markov structure is a better model of the real-world environments than domain variables with Markov structure. I don’t see why environments in the real-world would swap at random multiple times along a sequence of time points. For example, in the provided real-world experiment, ground-truth actions in the video in Figure S5 occur in blocks and they are autocorrelated. The weak-supervision of the action order used in this experiment also supports their autocorrelation.\n* According to the point above, the synthetic experimental setup seems unrealistic as it has randomly sampled domain variables.\n* I think the first contribution statement in Lines 59-61 is ambiguous: \"this is the first identifiability result that handles nonstationary time-delayed causally related latent temporal processes without prior knowledge of the domain variables\". What is meant by \"without the prior knowledge of domain variables\" is a bit vague, while the difference with the existing work [Song+23] seems to be the assumptions on the temporal dependence of the domain variables. [Song+23] assumes temporally-dependent domain variables and predicts the unknown domain variables with an HMM, while this work assumes temporally independent domain variables and predicts them from sequential observations.\n\n[Song+23]. Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. \"Temporally disentangled representation learning under unknown nonstationarity\" Neurips, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable questions and suggestions. Please kindly find our response below:\n\n> W1. Non-Testable Assumptions in Real-World Setup\n\nThank you for bringing attention to this common issue in the identifiability literature. We appreciate your insight. With your permission, we would like to add this statement to the paper to make this aspect more explicit.\n\n> W2. Assumption of Weakly Diverse Lossy Transition\n\n* To address the reviewer's question about the causal link: if there is a causal link $A \\to B$, it is possible to have cases where changing the value of $A$ doesn't change the distribution of $p(B|A)$. For example, $B = Relu(A) + \\epsilon_B$, we can clearly see that changing the value of $A$ within the interval $(-\\inf, 0)$ doesn't change the distribution of $B|A$. i.e., as long as there exists $a_1 \\neq a_2$ such that $p(B|A=a_1) \\neq p(B|A=a_2)$, then the causal link $A \\to B$ exists.\n* We thank the reviewer for the suggestion to further clarify the definition. We will add more illustrations and examples like the one above to make it clearer. However, since this is an assumption on the quantitative inference process, it may not be faithfully reflected in the graphical models.\n* Due to page limitations, we defer the discussion of the feasibility issue of this assumption to Appendix S1.4.5. In light of your comment, we feel it is better to avoid using terms like \"mild\" and \"realistic\" to prevent potentially overclaiming our contribution.\n\n> W3. Why Non-Markovian is Better? Real-World Autocorrelation\n\nFrom our understanding, the autocorrelation mentioned by the reviewer here refers to temporally dependent processes. However, it may not be reasonable to further assume that the states can be modeled by a single Markov chain. There are many situations where the Markov assumption is too restrictive. For example, by concatenating a state sequence generated from a Markov chain with a sequence generated from a temporally independent distribution, this simple scenario can already break HMM-based models (details discussed in the synthetic experiment lines 328-332), let alone more complex real-world cases.\n\n> W4. Synthetic Experiment Setup + W5. Assumptions on the Temporal Dependence of the Domain Variables\n\nWe would like to clarify that our method doesn't assume temporally independent domain variables; on the contrary, we allow a mix of temporally dependent and independent situations. As described in Appendix S2.1, in the synthetic setup the domain variables are not independently randomly sampled. Instead, it concatenates a state sequence generated from a Markov chain with a state sequence generated from a temporally independent distribution. Such a mixed setup is more complex compared with temporally dependent only, and the synthetic experiment also supports our claim.\n\n> W5. Without the Prior Knowledge + Q1. Aligned with Human Intuition\n\nWe defer the response to the global response as other reviewers also mentioned similar questions.\n\n> Q2. Some Missing Details for Experiments\n\n* For synthetic experiments, encoder $\\mathbf{g}^{-1}$ and decoder $\\mathbf{g}$ are implemented with a VAE structure, and transition networks $\\mathbf{m}_u$ are implemented with MLPs with a gating function to select the domain values.\n* Following existing work [Song+23], we randomly select 10% of the data for testing and leave the rest for training.\n* We use three random seeds for the training process, influencing the network parameter initialization and the optimization process.\n* For real-world experiments, we follow the settings and model architectures in the ATBA [Xu+24] baseline and report the testing results based on their existing dataset split, which is widely used in the action segmentation community. The results with random seeds also follow the convention in that domain, and details are provided in Appendix S3.4.\n* Please kindly let us know if our explanation properly addresses your concerns. By the way, all design details of the network and architecture can be found in the code (Appendix S3.3).\n\n> Q3. Metrics in Action Segmentation Task\n\nYes, the output of the model is the action class per frame in the video. As described in lines 359 and 391, the Mean-over-Frames (MoF) denotes the mean accuracy of the framewise action prediction. Following existing literature in action segmentation, we also report IoU and IoD.\n\n> Q3. How Weak Supervision is Provided\n\nWe follow the baseline method ATBA in using weak supervision. This information is provided as predefined ordered action patterns (so-called transcripts), and these transcripts are used in the boundary alignment process by lowering the score for candidate boundary assignments that violate the transcript order. We refer the reviewer to Sec 3.4 of [Xu+24] for detailed design.\n\n> Q3. Why NCTRL is Not in the Action Segmentation Task\n\nWe noticed there are already HMM-based methods in the baselines, and we have lines 374-394 directly discussing this line of methods. However, we agree that adding NCTRL for comparison makes the argument more convincing. The updated results in CrossTask can be found in the table below. We can see that since the Markov assumption may not be fullfilled by the transition of the action in real-world setting, the NCTRL cannot solve this problem well.\n\n|Method|MoF|IoU|IoD|\n|---|---|---|---|\n|NCTRL|50.7|12.9|23.3|\n|CtrlNS|**54.0**|**15.7**|**23.6**|\n\n> Q4. Sparsity Loss\n\nThe parameters of transition estimation functions refer to the neural networks we used to estimate the transition functions $\\mathbf{m}_{u}$. The sparse loss, as indicated in Eq. 15, is approximated using the $L_2$ norm of the parameters of those neural networks.\n\n[Xu+24] Xu. Angchi, et al. \"Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment.\" CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3AEiLusMvm", "reviewer": "Reviewer_1QG3", "summary": "The paper focuses on causal temporal representation learning for non-stationary time series. It adopts a sparse transition assumption, aligned with intuitive human understanding, and presents identifiability results from a theoretical perspective. Based on the theoretical result, the authors introduce a novel framework, Causal Temporal Representation Learning with Nonstationary Sparse Transition (CtrlNS), designed to leverage the constraints on transition sparsity and conditional independence to reliably identify both distribution shifts and latent factors. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in recovering latent variables and domain indices.", "questions": "Please see the Weaknesses.", "limitations": "The authors have appropriately discussed the limitations and societal impacts of the model.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper focuses on the issue of causal discovery in nonstationary time series, which is a fascinating and significant area of research.\n\n2.The authors claim to have proposed a method that establishes identifiability of nonstationary nonlinear ICA for general sequence data without prior knowledge of domain variables, which is an extremely challenging problem.\n\n3.The authors have conducted experiments on both synthetic and real-world datasets, validating the effectiveness of the model, and have appropriately discussed the limitations of the model.", "weaknesses": "1.The authors lack discussion of some existing work, for example, the paper [1] also demonstrates identifiability by clustering nonstationary spatiotemporal data into different domains without prior knowledge. It is essential for authors to further discuss the similarities and differences between the two papers and alleviate my concerns about the novelty of this work.\n\n2.In lines 259-261 of this paper, the authors mention ensuring the invertibility of the modeled mixing function through reconstruction loss, which is insufficient. Invertible Neural Networks (INNs) should be employed to ensure invertibility, as there will inevitably be errors in the reconstruction process.\n\n[1] Zhao, Yu, et al. \"Generative Causal Interpretation Model for Spatio-Temporal Representation Learning.\" Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable questions and suggestions. Please kindly find our response below:\n\n> Related Work: GCIM [1]\n\nWe appreciate the reviewer for highlighting the additional related work, GCIM. We will include this work in the related work section. Regarding the comparison, both works explore finding latent causal representations from observational data only and handle the nonstationary setting with unobserved domain variables. However, there are several differences between our work and GCIM:\n\n* First, from a theoretical aspect, GCIM only shows the identifiability of latent variables $\\mathbf{z}$ under the condition that the domain variables $u$ are observed. In the estimation part, as described in Section 3.2 of GCIM, it uses a Domain Adapter to empirically estimate domain variables, but the identifiability of domain variables is not addressed in their proof. One of our main contributions is to provide the identifiability result for the domain variables, clearly differentiating our work from GCIM.\n\n* Second, regarding the problem setting, as stated in Eq. 1 of [1], GCIM assumes the same causal structure of the transition function but with different nonstationary noise terms in different domains, which is similar to the setting in LEAP [2]. In contrast, our setting involves i.i.d. noise terms with different transition functions for each domain.\n\n> Invertible Neural Networks\n\nWe sincerely thank the reviewer for their valuable feedback and insightful suggestions. We fully agree that invertible neural networks, such as flow-based methods, are theoretically more aligned with our approach. In our experiments, we adopted the reconstruction loss-based method for the following reasons:\n\n* In our experimental setup, using reconstruction loss already provides strong identifiability results (MCC>0.95), and it is widely used in the identifiability literature, such as in [3,4].\n* During our experiments, we found that flow-based methods are usually less efficient and typically take longer to converge. Since our main contribution is to address the challenge of unknown domain variables, this choice is orthogonal to our theoretical contribution. Therefore, we followed the existing work for the design of the estimation for the mixing function.\n* As also mentioned in [3], the reconstruction loss-based framework can definitely be extended to flow-based methods, especially in environments where invertibility is a critical issue and computation is not a top priority in the estimation process.\n\nWe have included this discussion in the updated version of our paper and once again thank the reviewer for highlighting this important point.\n\n[2] Yao, Weiran, et al. \"Learning Temporally Causal Latent Processes from General Temporal Data.\" International Conference on Learning Representations, 2022. openreview.net/forum?id=RDlLMjLJXdq.\n\n[3] Zhang, Kun, et al. \"Causal Representation Learning from Multiple Distributions: A General Setting.\" 2024. arXiv, https://arxiv.org/abs/2402.05052.\n\n[4] Song, Xiangchen, et al. \"Temporally Disentangled Representation Learning under Unknown Nonstationarity.\" NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ytRYi6n37p", "reviewer": "Reviewer_6hAo", "summary": "Most existing causal temporal representation learning models either assume the domain variables to be observable, or have a Markov prior over them. This paper first comes up an identifiability theory for sequential data affected by non-stationary latent causal processes under unknown distributional shifts. In particular, a novel causal temporal representation learning method is developed to identify the latent causal variables with time-delayed causal relations. Moreover, without relying on prior knowledge on distributional shifts, the domain variables can be inferred from sequential observations. The final experiments on both synthetic and real-world action segmentation data, demonstrate the superior performance of the novel method in estimating domain variables and latent causal processes.", "questions": "Except the action segmentation tasks, can the authors provide more real applications including healthcare and finance, as mentioned in the boarder impacts.", "limitations": "Yes, the paper discusses the potential negative societal impact of the work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nIt is the first attempt to develop causal temporal representational learning to identify both the domain variables and the latent causal processes from sequential observations, without relying on distributional or structural knowledge.\n\nQuality:\nThe theorectical aspects of the paper look very sound, and absolutely pass the acceptance bar of the NeurIPS.\n \nClarity:\nOverall, the paper is well written. Both the target problems and the main techniques are well presented!\n \nSignificance:\nThe developed identifiability theory for domain variables and latent causal processes from sequential observations under unknown distributional or structural prior knowledge, may have impacts for advancing the relevant causal representation learning methods.", "weaknesses": "Throughout the paper, I did not find the details about how to set up the parameters of the neural nets for the novel CTRL framework, which could be crucial for the others to pick up the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our work and contribution, and we also thank you for providing valuable questions and suggestions. Please kindly find our response below:\n\n> W1. Parameter Settings\n\nWe appreciate the reviewer for raising this suggestion, which has improved our readability. In terms of the neural network design, our CtrlNS model is based on the baseline implementation of NCTRL. Specifically, we replaced the HMM module with our sparse transition module while keeping the encoder and decoder modules consistent with the baseline. The sparse transition module is implemented using MLPs to create gating functions in conjunction with the transition function employed in our methods. More details can be found in the codebase provided in Appendix S3.3. In light of the suggestion, we have included such details in the experiments section of the updated version.\n\n> Q1. Additional Real Applications (Healthcare and Finance)\n\nWe highly appreciate your valuable suggestion about applying our method to better verify its effectiveness. Beyond the action segmentation task, our proposed method can also be used in sensory data in the healthcare domain. For example, Apple Watch sensor data can be utilized to automatically detect health condition changes and remind the user to seek medical help if the domain variable indicates a risky health condition. A similar design is also useful in the finance domain when monitoring market data to perform early detection of black swan events.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for providing valuable and insightful questions and suggestions on our work. We found that our claim of "without prior knowledge of domain variables" and the "motivation to align with human intuition" are mentioned by multiple reviewers. We give a comprehensive response here.

> Regarding Without Prior Knowledge of Domain Variables

We would like to thank reviewer Zo2n for summarizing that NCTRL assumes temporally-dependent domain variables and predicts the unknown domain variables with an HMM. Such an assumption on the distribution form of the domain variables is widely used in the literature when dealing with unobserved domain variables.

In contrast, our claim is that we do not need such information about the domain variables. In our case, the domain variables can be either temporally dependent or independent; it doesn't matter. As long as the transitions controlled by these domain variables are sufficiently different, we can still recover those domain variables from observational data. 

We thank both reviewers 3MKp and Zo2n for mentioning this aspect, and we have revised this claim to a more moderate statement: "doesn't rely on knowledge of the prior distribution of the domain variables" to avoid overclaiming our contribution. We hope this helps to clarify the message we want to convey to the audience.

> Regarding the Motivation of the Assumptions Aligning with Human Intuition

We thank reviewer Zo2n for raising this question and thank reviewer 3MKp for mentioning the comparison of assumptions with existing work. We leverage this opportunity to clarify and emphasize the central idea behind our work.

* We assume human observers perform the reasoning process with meaningful representation in their minds. Given that humans have this ability, our goal is to enable machines to also possess this capability. Under the scope of this work, we are exploring the decision boundary on the conditions under which a machine can act like a human to identify the domains from observational data.
* We aim to quantify the significance of such changes among different domains. Our assumption of at least one edge being different across domains is just a sufficient condition in which the changes are significant enough to establish identifiability for machines.
* It is not a necessary condition; there are certainly cases where all edges are the same for two domains, but humans can still tell the difference.
* We partially explored this case in Sec 3.2 remark, where we use higher-order sparsity to achieve identifiability. We leave the exploration of the rest of the cases for future work and also encourage the community to join us in further exploring this decision boundary between machine perception and human perception.

Meta-review: The paper presents a causal temporal representation learning framework for identifying distribution shifts and latent factors in nonstationary time series without strong prior knowledge of domain variables. It introduces a sparse transition assumption and provides theoretical identifiability results for both domain variables and latent causal processes from sequential observations. 

Strengths:

+ Tackles an important problem of causal temporal representation learning under nonstationarity with sparse transition assumptions than prior work

+ Provides identifiability results

+ Evaluates the method on both synthetic and realistic tasks (action segmentation)

Weaknesses:

+ Some key assumptions (e.g. weakly diverse lossy transitions) can be unrealistic 

+ Lacks comparison to relevant related work (that may rely on different identification assumptions)
Decision: Accept (poster)