Title: Sample Complexity of Interventional Causal Representation Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causal representation learning

sample complexity

interventions

score-based methods

identifiability
Abstract: Consider a data-generation process that transforms low-dimensional _latent_ causally-related variables to high-dimensional _observed_ variables. Causal representation learning (CRL) is the process of using the observed data to recover the latent causal variables and the causal structure among them. Despite the multitude of identifiability results under various interventional CRL settings, the existing guarantees apply exclusively to the _infinite-sample_ regime (i.e., infinite observed samples). This paper establishes the first sample-complexity analysis for the finite-sample regime, in which the interactions between the number of observed samples and probabilistic guarantees on recovering the latent variables and structure are established. This paper focuses on _general_ latent causal models, stochastic _soft_ interventions, and a linear transformation from the latent to the observation space. The identifiability results ensure graph recovery up to ancestors and latent variables recovery up to mixing with parent variables. Specifically, ${\cal O}((\log \frac{1}{\delta})^{4})$ samples suffice for latent graph recovery up to ancestors with probability $1 - \delta$, and ${\cal O}((\frac{1}{\epsilon}\log \frac{1}{\delta})^{4})$ samples suffice for latent causal variables recovery that is $\epsilon$ close to the identifiability class with probability $1 - \delta$.

Sample Complexity of Interventional Causal Representation Learning

Abstract

Consider a data-generation process that transforms low-dimensional latent causallyrelated variables to high-dimensional observed variables. Causal representation learning (CRL) is the process of using the observed data to recover the latent causal variables and the causal structure among them. Despite the multitude of identifiability results under various interventional CRL settings, the existing guarantees apply exclusively to the infinite-sample regime (i.e., infinite observed samples). This paper establishes the first sample-complexity analysis for the finite-sample regime, in which the interactions between the number of observed samples and probabilistic guarantees on recovering the latent variables and structure are established. This paper focuses on general latent causal models, stochastic soft interventions, and a linear transformation from the latent to the observation space. The identifiability results ensure graph recovery up to ancestors and latent variables recovery up to mixing with parent variables. Specifically, O ((log 1 δ ) 4 ) samples suffice for latent graph recovery up to ancestors with probability 1 -δ , and O (( 1 ϵ log 1 δ ) 4 ) samples suffice for latent causal variables recovery that is ϵ close to the identifiability class with probability 1 -δ .

1 Introduction

The observed data generated in a wide range of technological, social, and biological domains has high-dimensional, complex, and often unexplainable structures. Nevertheless, sometimes, such complex structures can be explained by latent generating factors that often form lower-dimensional structures. The field of causal representation learning (CRL) is motivated by this premise and focuses on disentangling the causally -related latent generating factors that underlie a given high-dimensional observable dataset. In particular, given a high-dimensional dataset, the objective of CRL is to learn (i) the underlying latent causal variables and (ii) the causal relationships among the latent variables. The learned representations then provide an explainable structure for the observed data and facilitate informed reasoning for the downstream tasks [1].

Formally, CRL consists of a data generation and data transformation pipeline as follows. There exists a set of high-level latent variables Z ∈ R n that are causally related. The causal interactions are captured by a directed acyclic graph (DAG) G . The latent variables Z go through an unknown transformation g and generate the observed variables X ∈ R d , i.e., X = g ( Z ) . The objective of CRL is to use the observed variables X and learn the latent graph G and latent variables Z . The primary question facing CRL is identifiability , which refers to establishing the conditions under which recovering Z and G are possible.

Identifiability is known to be impossible without proper inductive biases or additional supervision [2, 3]. One approach to address this issue is performing interventions - leading to interventional CRL - which has gained significant recent attention [4, 5, 6, 7, 8, 9, 10]. Specifically, interventions on latent causal variables create additional statistical diversity in the observed data. Accessing to enough interventional environments enables identifiability [1]. Despite the recent advances in establishing identifiability guarantees for interventional CRL, all the existing guarantees focus on the asymptotic infinite-sample regime, where one assumes access to an infinite number of observable samples X .

In this paper, we establish conditions for finite-sample non-asymptotic identifiability and performance guarantees for interventional CRL. We focus on CRL under the general (non-parametric) latent causal models, soft interventions, and linear transformations from the latent to the observation space. This setting is well-studied in the infinite-sample regime [4, 5, 7, 11], where the existing identifiability results show that the causal graph can be recovered up to ancestral nodes, and latent variables can be learned up to mixing with their ancestors. In this paper, we provide the probabilistic finite-sample counterparts of these identifiability guarantees and establish the first sample complexity analysis for interventional CRL in the finite-sample regime.

Our CRL approach falls in the category of score-based CRL [5], based on which our sample complexity analysis consists of two steps. In the first step, we delineate a general sample complexity for any desired consistent score estimator. Subsequently, we specialize these general results by adopting the reproducing kernel Hilbert space (RKHS)-based score estimator [12]. We establish the following identifiability guarantees for any desired pair of constants ϵ, δ ∈ R + .

Latent graph recovery: Using RHKS-based score estimator, O ((log 1 δ ) 4 ) samples suffice to recover the transitive closure of the latent graph with probability at least 1 -δ .

Latent variables recovery: Using the same score estimator, O (( 1 ϵ log 1 δ ) 4 ) samples suffice to ensure that the mean squared error in the estimated latent causal variables is at most ϵ 2 with probability at least 1 -δ .

Dependence on model dimension: We further establish the explicit and implicit dependence of sample complexity on the dimensions of the latent and observable spaces, n and d , respectively. The precise characterizations of these expressions involve additional model-dependent constants specified in Section 5.

Improved guarantees: Finally, we note that our latent variables recovery result, where we show recovery up to mixing with parents , which improves upon the results in existing literature that guarantee recovering up to mixing with ancestors .

Methodology. We offer novel finite-sample CRL algorithms and characterize the sample complexity guarantees achievable by these algorithms. We design our algorithms using the properties of score functions (i.e., the gradient of the log density). This approach is inspired by the score-based CRL framework [5, 8]. However, the algorithms are significantly different, led by the need to integrate finite-sample score estimation routines and their imperfections. In our analysis, first, we provide sample complexity upper bounds for a score-based framework that uses any desired consistent score difference estimator. Then, we adopt a specific score estimator [12] and provide explicit sample complexity guarantees.

Related work. All the existing studies on interventional CRL focus on the infinite-sample regime. We briefly review the studies that adopt a similar CRL model, i.e., linear transformations and singlenode soft interventions. In this setting, complete identifiability results for linear latent models are established in [7]. Similar results are shown for nonlinear latent causal models in [4, 11] and for linear non-Gaussian models in [13]. In the most closely related setting to ours, identifiability up to ancestors is shown to be possible using soft interventions on general latent causal models [8]. Other studies on interventional CRL include using do interventions for polynomial transformations [6, 14] and hard interventions for general transformations [10, 8]. On a partially related problem, error rates for using score functions on observed causal discovery are provided in [15].

Notations. For n ∈ N , we define [ n ] ≜ { 1 , . . . , n } . Vectors are represented by lower case bold letters, and element i of vector v is denoted by v i . Matrices are represented by upper case bold letters, and we denote row i and column j of matrix A by A i, : and A : ,j , respectively. The row permutation matrix for permutation π of [ n ] is denoted by P π . Random variables and their realizations are presented by upper and lower case letters, respectively. We denote the Moore-Penrose pseudoinverse

of a matrix A by A † . Given a symmetric matrix A ∈ R d × d , we denote the vector of eigenvalues of A ordered in ascending order by λ ( A ) ∈ R d and the matrix of eigenvectors by Q ( A ) ∈ R d × d such that A = Q ( A ) · diag ( λ ( A )) · Q ( A ) ⊤ . For any matrix A , we denote the rank, column, and null spaces of A by rank ( A ) , col ( A ) , and null ( A ) , respectively.

2 Finite-sample data-generation process

The existing studies on analyzing the identifiability guarantees of CRL inevitably necessitate the availability of an infinite number of samples as otherwise, perfect identifiability is rendered impossible. In this paper, our objective is identifiability analysis assuming only a finite number of samples are available - a significant departure from the infinite-sample regime. In this section, we specify a general data-generating process consisting of a latent causal space and an unknown linear transformation that maps the latent variables to observed variables.

Latent causal model. We have a latent space of n causally related random variables. We denote the latent causal variables by Z ≜ [ Z 1 , . . . , Z n ] ⊤ . The causal relationships among the variables of Z are represented by a directed acyclic graph G with n nodes where the i -th node represents Z i . We denote the parents, children, and ancestors of a node i ∈ [ n ] in G by pa( i ) , ch( i ) , and an( i ) , respectively. We denote the probability density function (pdf) of Z by p Z , which is assumed to be well-defined without any zeros over lower-dimensional manifolds and to have full support on R n . Given the DAG G , p Z factorizes according to

We assume that the conditional pdfs { p i : i ∈ [ n ] } are continuously differentiable with respect to z . We call a permutation π = ( π 1 , . . . , π n ) of [ n ] a valid causal order if for all i, j ∈ [ n ] , i ∈ pa( j ) implies π i < π j . Without loss of generality, we assume that (1 , . . . , n ) is a valid causal order.

Transformation model. The latent variables Z are mapped to the observed variables X ≜ [ X 1 , . . . , X d ] ⊤ through an unknown linear transformation G ∈ R d × n , where d ≥ n . Specifically,

where G has full column rank. We denote the pdf of X by p X . Note that owing to the generation process of X , the pdf p X is supported on an n -dimensional subspace embedded in R d . We denote the L 2 norm of any function f with finite variance under p X by ∥ f ∥ 2 p X ≜ E p X ∥ f ( x ) ∥ 2 2 .

Intervention model. We consider CRL under interventions, and in particular focus on soft interventions, as the most general form of interventions. Hence, in addition to the observational model specified by (2), we consider n single-node interventional environments {E m : m ∈ [ n ] } . We assume that the node intervened in the environment E m is unknown and denote it by I m . Leaving one node unintervened renders identifiability impossible [7]. Hence, we inevitably have { I m : m ∈ [ n ] } = [ n ] .

Applying a soft intervention on node i changes its causal mechanism specified by the observational conditional pdf p i ( z i | z pa( i ) ) to a distinct interventional conditional pdf q i ( z i | z pa( i ) ) . The conditional pdfs { q i : i ∈ [ n ] } are assumed to be continuously differentiable with respect to z . Subsequently, the pdf of Z in the interventional environment E m , denoted by p m , factorizes according to

̸

To distinguish the observational and interventional data, we denote the latent and observed random variables in environment E m by Z m and X m , respectively. Interventions do not alter the transformation matrix G , indicating that similarly to (2), we have X m = G · Z m . Throughout the paper, we adopt the convention that E 0 refers to the observational environment.

Score functions. The score function associated with a pdf is defined as the gradient of its logarithm. We denote the score functions associated with the observational distributions of Z and X by s Z and s X , respectively, i.e.,

Similarly, we denote the score functions associated with the distributions of Z and X in environment E m for m ∈ [ n ] by s m Z and s m X , respectively, i.e.,

In our algorithm and analysis, we will analyze the score variations across different interventions. For this purpose, we define the score difference functions of Z and X between environments E m and E 0 as

To ensure that interventions affect the target node and its parents distinctly, we adopt the following assumption. This assumption holds for a large class of models, e.g., additive noise models under stochastic hard interventions [5, Lemma 2].

Assumption 1 ([5, Assumption 1]) . For any m ∈ [ n ] , and for all k ∈ pa( I m ) , the term ( [ d m Z ] k / [ d m Z ] I m ) is not a constant function of z .

Finite sample data. We assume that we have only N data samples under each of the environments. We denote the N samples of Z m and X m by

Accordingly, for each sample j , we concatenate all latent and observational samples under different environments to create sample matrices

Finally, we define the set of samples X N ≜ { X j : j ∈ [ N ] } and Z N ≜ { Z j : j ∈ [ N ] } .

3 Finite-sample identifiability objectives

The CRL framework specified in Section 2 has two objectives: use the N observational samples X N = { X j : j ∈ [ N ] } to recover the causal graph G and the latent causal variables Z N = { Z j : j ∈ [ N ] } . In this section, we formalize the inference rules and their associated fidelity metrics for recovering G and Z N .

Latent graph recovery. Define G as the set of all DAGs on n nodes. We define ˆ G as a generic estimator of the latent graph G , i.e.,

To quantify the accuracy of the estimate ˆ G ( X N ) , we provide the following probably approximately correct (PAC) guarantee, which is the probabilistic counterpart of the standard identifiability up to ancestor definition in the interventional CRL literature [5, 7, 11].

Definition 1 ( δ -PAC graph recovery) . Graph estimate ˆ G ( X N ) achieves δ -PAC latent graph recovery if, with probability at least 1 -δ , the transitive closures of ˆ G ( X N ) and G are isomorphic.

Latent variables recovery. We investigate a two-step estimator for the latent variables Z N . First, we define a generic estimator for the pseudoinverse of G as

Then, given an estimate H ( X N ) for G † , we estimate { Z j : j ∈ [ N ] } according to

We provide the following definition to quantify the fidelity of these estimates with respect to the ground truth variables.

Definition 2 ( ( ϵ, δ ) -PAC variables recovery) . The estimate H ( X N ) achieves ( ϵ, δ ) -PAC latent variables recovery if the estimated causal variables { ˆ Z j : j ∈ [ N ] } satisfy

where for all i ̸∈ pa( j ) , C pa satisfies ( C pa ) i,j = 0 , and, with probability at least 1 -δ , we have ∥ C err ∥ 2 ≤ ϵ .

We note that Definition 2 is the probabilistic counterpart of the standard identifiability up to ancestor definition in interventional CRL literature.

Noisy score model. The score-based CRL framework uses properties of score function differences to build the estimators for the latent graph and variables. Therefore, when we only have access to finite data, we need to estimate the score functions. We denote a generic score function estimator for environment E m by

When the dependence is clear from the context, we will drop the explicit dependence of ˆ s m X on X N . Similarly to (6), we define the estimated score difference functions

In this paper, we focus on consistent score difference estimators { ˆ d m X : m ∈ [ n ] } , i.e., they satisfy convergence in probability. Specifically, for any ϵ, δ > 0 , there exists N ( ϵ, δ ) ∈ N such that

Notably, we note that many score estimators are known to be consistent [12, 16, 17, 18].

4 Finite-sample CRL algorithms

In this section, we design finite-sample interventional CRL algorithms through which we establish finite-sample identifiability guarantees as well as bounds on the associated sample complexities. Our algorithms fall within the score-based category of CRL algorithms [5]. The main intuition of this framework is that score functions in the observed space contain all the information needed for recovering the latent graph G and the inverse transform G † . Specifically, two metrics are pivotal for retrieving the latent space:

Score differences: As shown in Lemma 1, the nonzero entries of the latent score differences d m Z encode the graph structure and the observed score differences d m X are generated using the inverse transform G † .

Score difference correlations: As shown in Lemma 2, the column space of the correlation matrix of the score differences contains crucial information about the latent graph and the inverse transform, which we will leverage to form estimates for the latent graph G and the inverse transform G † .

To proceed, for m ∈ [ n ] we denote the correlation matrices of d m X and ˆ d m X by

Next, we formalize two critical properties of score differences and their correlation matrix. Specifically, the following lemma states that the sparsity pattern of the latent score differences d m Z exposes the structure of the latent graph, and the observed score differences d m X preserve this information through the pseudoinverse of G .

Lemma 1 ([5]) . Score function differences in the latent space satisfy, for all m ∈ [ n ] ,

Furthermore, the score differences in the observed domain are given by

The next lemma specifies that the structure of the column space of correlation matrix R m X is heavily constrained by the graph structure and the inverse transformation G † .

While these two properties enable recovering the latent graph and variables, the ground truth score functions d m Z ( z ) and correlation matrix R m X are unknown. We only have access to their noisy counterparts, as estimated from the observed data. In our algorithms, we use methods with soft decision rules that can counter the effects of the errors introduced by the score estimation procedure. These are facilitated by the following three key definitions.

Algorithm 1 Causal order estimation

Algorithm 2 Graph estimation

Definition 3 (Approximate column space) . We define the η -approximate column space of a positive semidefinite matrix A ∈ R d × d , denoted by col ( A ; η ) , as the span of the eigenvectors of A associated with the eigenvalues that are strictly greater than η .

Definition 4 (Approximate null space) . We define the η -approximate null space of A , denoted by null ( A ; η ) , as the orthogonal complement of col ( A ; η ) .

Definition 5 (Approximate subspace orthogonality) . We say two subspaces A and B with orthonormal bases A and B are γ -approximately orthogonal, denoted by A ⊥ γ B , if ∥ B ⊤ A ∥ 2 ≤ γ .

In these definitions, setting η = γ = 0 yields the standard definitions of column and null spaces and orthogonality. Next, we describe our algorithms for latent graph recovery (Algorithms 1 and 2) and latent variables recovery (Algorithm 3). We note that the algorithms for latent graph and latent variable recovery algorithms are fully decoupled, enabling the independent recovery of the graph and the latent variables.

Algorithm 1 - Causal order estimation. In this algorithm, we estimate a permutation π of [ n ] such that I ◦ π is a valid causal order. This serves as an intermediate step for estimating the latent graph. For this purpose, we use the noisy correlation matrices { ˆ R m X : m ∈ [ n ] } to identify the leaf nodes with no children. In particular, the key property is that when I k is a leaf node, by carefully selecting the threshold η , the approximate column space of the term ( ∑ m ∈ [ n ] ˆ R m X -ˆ R k X ) has dimension n -1 . Precisely, with high probability, we have

After finding a leaf node, we iteratively identify the youngest node among the remaining set of nodes. Leveraging this, we construct the permutation π , which consists of nodes ordered from the eldest to the youngest. In Lemma 3, we show that I ◦ π is a valid causal order with high probability.

Algorithm 2 - Latent graph estimation. In Algorithm 2, we use the causal order found in Algorithm 1 and correlation matrices { ˆ R m X : m ∈ [ n ] } to form a graph estimate ˆ G . We build this graph iteratively by considering a candidate edge π t → π j for all possible ( t, j ) pairs starting from a leaf node t . The key property that we leverage is that we can form two subsets V t and M t,j of [ n ] using π ,

Algorithm 3 Inverse transform estimation

1: Input: ˆ R m for all m ∈ [ n ] , η ≥ 0 2: H ← 0 n × d 3: for t ∈ (1 , . . . , n ) do 4: C ← col ( ˆ R π t ; η ) 5: H π t , : ← v for v ∈ C and ∥ v ∥ 2 = 1

6: Return H

t , and j such that, for sufficiently small γ , with high probability the following approximation holds.

We check each edge π t → π j and add the detected edges to the estimated graph ˆ G . In Theorem 1, we show that this procedure guarantees a PAC latent graph recovery.

Algorithm 3 - Inverse transform estimation. In Algorithm 3, we build our inverse transform estimate H one row at a time. The key property we use is that for any m ∈ [ n ] and unit vector v ∈ col ( ˆ R m X ) , the following error term is small with high probability

In other words v ⊤ · X is approximately equal to Z I m up to mixing with { Z i : i ∈ pa( I m ) } , which conforms to our latent variables recovery objective. Note that in the noise-free setting, an exact equality holds due to Lemma 2 and property G † · G = I n . Based on this observation, in our algorithm, we construct our inverse transform estimate by setting row m of H to a unit vector v ∈ col ( ˆ R m X ) for all m ∈ [ n ] . In Theorem 2, we show that Algorithm 3 achieves a PAC latent variables recovery.

5 Sample complexity analysis

In this section, we analyze the sample complexity of our CRL algorithm to establish an achievable sample complexity for CRL.

Threshold selection. A critical step in our algorithm designs is choosing thresholds η and γ specified in Definitions 3-5. These thresholds determine the approximate ranks of noisy matrices and approximate orthogonality between subspaces, respectively. Specifically, we select η such that the approximate rank of ˆ R m X tracks that of R m X with high probability. To proceed, we denote the minimum nonzero eigenvalue among arbitrary R m X sums by

̸

where λ ( A ) ∈ R d denotes the vector of eigenvalues of A . In our algorithms, we let η ∈ (0 , η ∗ ) and show that this choice ensures that eigenvalues of the null and column spaces can be separated via η . For the approximate orthogonality test in Definition 5, we show that γ ∗ defined below serves as a bound on how close the non-orthogonal column spaces of R m X sums become orthogonal.

Similarly, we let γ ∈ (0 , γ ∗ ) in our algorithms. For any choice of η and γ , we define η ∗ ≜ min { η, η ∗ -η } and γ ∗ ≜ min { γ, γ ∗ -γ } . We note that we do not have to know η ∗ and γ ∗ a priori, and can include routines to estimate them in practice. These estimates do not have to be highly accurate since even rough estimates suffice to choose reliable thresholds η and γ . Specifically, based on estimates for η ∗ and γ ∗ , we can choose 'safe' thresholds for η and γ , e.g., one-fourth of the estimates for η ∗ and γ ∗ , so that (i) with high probability we satisfy the requirement η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) , and (ii) we can avoid collecting excessive samples and compromising the sample complexity bounds. We provide the details of constructing such estimates of η ∗ and γ ∗ in Appendix G.

Sample complexity results. We provide two sets of sample complexity analyses for the latent graph and the latent variables. We estimate the latent graph in two steps: (i) causal order estimation

(Algorithm 1) and (ii) latent graph estimation (Algorithm 2). Algorithm 1 only employs approximate rank tests. Therefore, we first show that the approximate rank of the sum of correlation matrices { ˆ R m X : m ∈ [ n ] } tracks that of { R m X : m ∈ [ n ] } with high probability. To proceed, we define the following constants that appear repeatedly in our analysis:

Lemma 3. Let η ∈ (0 , η ∗ ) . For any δ > 0 , N rank ( δ ) samples suffice to ensure that with probability at least 1 -δ ,

where

Note that the function N ( · , · ) is an inherent property of the score difference estimator specified in (15), and it is monotonically decreasing in its second argument. Using this lemma, we can show that Algorithm 1 returns a permutation π such that I ◦ π is a valid causal order with high probability.

Lemma 4. Let η ∈ (0 , η ∗ ) . Under Assumption 1, for any δ > 0 , N rank ( δ ) samples suffice to ensure that with probability at least 1 -δ , I ◦ π is a valid causal order, where π is the output of Algorithm 1.

Next, we show that Algorithm 2 achieves latent graph recovery with high probability.

Theorem 1 (Sample complexity - Graph) . Let η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) . Under Assumption 1, for any δ > 0 , N G ( δ ) samples suffice to ensure that the output ˆ G ( X N ) collectively generated by Algorithms 1 and 2 satisfies δ -PAC graph recovery, where

We note that constants β and β min are sample-independent. Then, (27) specifies the sample complexity as a function of the hyperparameters η, γ , target reliability δ ∈ (0 , 1) , as well as the latent dimension n . Next, we state the complementary result for recovering the latent variables.

Theorem 2 (Sample complexity - Variables) . Let η ∈ (0 , η ∗ ) . For any ϵ > 0 and δ > 0 , N Z ( ϵ, δ ) samples suffice to ensure that the output H ( X N ) of Algorithm 3 satisfies ( ϵ, δ ) -PAC causal variables recovery, where

Theorems 1 and 2 collectively specify an extent of identifiability achievable in the finite-sample regime. Importantly, we note that since we can establish ( ϵ, δ ) -PAC identifiability guarantees for any vanishing ϵ, δ > 0 using finite samples, Algorithms 2 and 3 are consistent estimators of the latent graph and the inverse transformation up to the corresponding equivalence classes.

Finally, we note that the latent variables are recovered up to mixing with parents , as specified in Theorem 2. This result is a refinement of the existing latent variable recovery results under soft interventions in existing literature [5, 7], which recover the latent variables up to mixing with ancestors .

RKHS-based score estimator. So far, we have characterized the sample complexity for any consistent score difference estimator. Next, we specialize the results by adopting the RKHS-based score estimator of [12]. We adopt this particular choice since it has known non-asymptotic sample complexity properties. To our knowledge, it is the only score estimator equipped with such a guarantee. To use the sample complexity of this score estimator in our paper, we first show that it is consistent in the sense of (15). For the formal statement of the following property and its attendant assumptions, we refer to Appendix E.

Lemma 5 (Informal) . Assume that s X and s m X satisfy the conditions of the RKHS-based score estimator in [12]. Then, for any given δ and ϵ , the convergence specified in (15) holds when

where κ and C are sample independent constants that depends only on p X , p m X for m ∈ [ n ] and the structure of the RKHS.

We use Lemma 5 to customize the general sample complexity bounds in Theorems 1 and 2 to the RKHS-based score estimator. For this purpose, we first provide our sample complexity result for recovering the latent graph.

Theorem 3 (RKHS-based sample complexity - Graph) . Let η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) . Under Assumption 1 and the conditions of Lemma 5, N G ( δ ) samples suffice to ensure that ˆ G ( X N ) collectively generated by Algorithms 1 and 2 satisfies δ -PAC graph recovery, where

Remark 1 (RKHS-based error bound - Graph) . Theorem 3 implies that using N samples, the output ˆ G ( X N ) of Algorithms 1 and 2 satisfies δ G ( N ) -PAC graph recovery, where

Similarly, we leverage Lemma 5 to specialize the general sample complexity in Theorem 2 to the RKHS-based score estimator.

Theorem 4 (RKHS-based sample complexity - Variables) . Let η ∈ (0 , η ∗ ) . Under the conditions of Lemma 5, N Z ( ϵ, δ ) samples suffice to ensure that the output H ( X N ) of Algorithm 3 satisfies ( ϵ, δ ) -PAC causal variables recovery, where

and

We note that the results in Theorems 3 and 4 show a more explicit dependence of the sample complexity on the latent space dimension n . We observe that the sample complexity of latent graph recovery explicitly depends on δ and the latent dimension n according to O (( n log n δ ) 4 ) . Similarly, the sample complexity of latent variables recovery explicitly depends on ϵ , δ , and the latent dimension n according to O (( √ n ϵ log n δ ) 4 ) . We note that constants β , β min , η ∗ , and γ ∗ are model parameters, and their scaling behavior in terms of n and d are investigated numerically in the next section.

6 Experiments

In this section, we perform numerical assessments of our analyses to provide complementary insight into the sample complexity results of Section 5. Specifically, we evaluate the variations of the model constants with respect to problem dimensions n and d . Furthermore, we also evaluate performance variations of the finite-sample algorithm in terms of the variations in the score estimation error. For this purpose, we focus on the mean squared error (MSE) of the score estimator, specified by E ∥ ˆ d m X -d m X ∥ 2 p X . These evaluations facilitate a better understanding of the properties of the CRL problem that are not explicit in our theoretical queries. 2

Experimental details. Weconsider problem dimensions n ∈ { 3 , 5 , 10 } and d ∈ { n, 15 } and generate G using Erd˝ os-Rényi model with density 0 . 5 on n nodes. We adopt linear Gaussian models as the latent causal model. We consider N ∈ { 10 2 . 5 , 10 3 , 10 3 . 5 , 10 4 , 10 4 . 5 , 10 5 } samples, and generate 100 latent models for each triplet ( N,n,d ) . We generate N samples of Z from each environment for each latent model. Transformation G ∈ R d × n is randomly sampled under full-rank and bounded condition number constraints, and the observed variables are generated as X = G · Z . Due to the Gaussian structure of Z , the observed variables X also have multivariate Gaussian distributions with

(a) Constants vs. problem dimension.

(b) Graph recovery vs. score MSE, d = 15

1.00

n=3

0.75

n=5

recover

x

n=10

0.50

Graph

0.25

0.00

X

xx

10

-2

10

=1

10°

0

10

1

ScoreestimationMSE

10

Constants

10

1

10

1

β

10

-3

βmin

4

6

8

10

Numberofnodes

score function s X ( x ) = -Θ · x , where Θ is the precision matrix of X . Therefore, we estimate s X using the sample precision matrix estimated from N samples, ˆ Θ N , as ˆ s X ( x ; X N ) = -ˆ Θ N · x .

Results. The sample complexity results depend on dimension n and d through their explicit presence as well as their implicit effect on model-dependent constants β , β min , η ∗ , and γ ∗ , as established in Theorems 1 and 2. First, we observe that all these parameters are independent of d . This is also expected theoretically since we can project down the d dimensional observed space to the n dimensional col ( G ) . In Figure 1a, we illustrate the dependence of these constants on data dimensions n and d . We observe that γ ∗ remains constant, while β and β min are decreasing as n increases, but the decay is not steep. On the other hand, we observe that η ∗ decays exponentially with n . γ ∗ is implicitly controlled by the condition number, which is kept bounded. We recall that β min is the minimum among {∥ d m X ∥ p X : m ∈ [ n ] } , and similarly, β is the inverse of the maximum of {∥ d m X ∥ p X : m ∈ [ n ] } , so they are expected to become smaller as n increases. Finally, for η ∗ , we note that the definition in (22) is an order statistic among exponentially many, 2 n , subsets.

Next, we note that all the imperfections in the graph and variable recovery are due to the imperfections in score estimates. In fact, this is the bottleneck in our decisions: the key impact of finite samples versus infinite samples is the degradation in the quality of score estimates. To have a direct insight into how the score estimation noise power translates into decision imperfection, we assess the quality of graph recovery success rate versus varying degrees of mean score error of score difference estimates E [ ∥ ˆ d m X -d m X ∥ 2 p X ] for different triplets ( N,n,d ) . In Figure 1b, each data point corresponds to a different sample size N , and the legend entries { 3 , 5 , 10 } denote the latent dimension n . This figure illustrates the scatter plot of the error probability δ versus MSE for different parameter combinations and the corresponding local regression curves. It is observed that in the low MSE regime, 1 -δ decays linearly with respect to log E [ ∥ ˆ d m X -d m X ∥ 2 p X ] and it plateaus as the MSE increases. This trend is due to our algorithms' high sensitivity to errors in estimating approximate matrix ranks.

7 Conclusion

In this paper, we have established the first sample complexity results for interventional causal representation learning. Specifically, we have characterized upper bounds on the sample complexity of latent graph and variables recovery in terms of the finite sample performance of any consistent score difference estimator. We have, furthermore, adopted a particular score estimator [12] to derive explicit sample complexity statements.

Our sample complexity results are given for partial identifiability (up to ancestors) in the soft intervention and linear transformation setting. Establishing sample complexity results for perfect identifiability via hard interventions or considering general transformations are important future directions.

References

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, May 2021.

Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks , 12(3):429-439, April 1999.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proc. International Conference on Machine Learning , Long Beach, CA, June 2019.

Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Scorebased causal representation learning with interventions. arXiv:2301.08230 , 2023.

Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Scorebased causal representation learning: Linear and general transformations. arXiv:2402.00849 , 2024.

Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In Proc. International Conference on Machine Learning , Honolulu, Hawaii, July 2023.

Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In Proc. International Conference on Machine Learning , Honolulu, Hawaii, July 2023.

Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning. In Proc. International Conference on Artificial Intelligence and Statistics , Valencia, Spain, May 2024.

Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, December 2023.

Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Keki´ c, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, December 2023.

Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, December 2023.

Yuhao Zhou, Jiaxin Shi, and Jun Zhu. Nonparametric score estimators. In Proc. International Conference on Machine Learning , virtual, July 2020.

Jikai Jin and Vasilis Syrgkanis. Learning causal representations from general environments: Identifiability and intrinsic ambiguity. arXiv:2311.12267 , 2023.

Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Identifying linearly-mixed causal representations from multi-node interventions. In Proc. Causal Learning and Reasoning , pages 843-867, Los Angeles, CA, April 2024.

Zhenyu Zhu, Francesco Locatello, and Volkan Cevher. Sample complexity bounds for scorematching: Causal discovery and generative modeling. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, December 2023.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4):695-709, July 2005.

Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proc. Uncertainty in Artificial Intelligence , virtual, August 2020.

Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv:2402.07747 , 2024.

Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis , 7(1):1-46, 1970.

Sample Complexity of Interventional Causal Representation Learning: Appendices

Table of Contents

A Proof structure

We structure the proofs of our main results as follows.

Appendix B - Infinite sample guarantees: First, we prove identifiability results in the noise-free setting, that is, using the ground truth R m X matrices instead of using the estimated ˆ R m X . These intermediate results lay the groundwork for the proofs for the finite sample setting.

Appendix C - Bounded noise guarantees: Next, we show that if estimation error for the score difference functions, ∥ ˆ d m X -d m X ∥ p X , is bounded, then the noise-free identifiability guarantees transfer to the noisy setting.

Appendix D - Sample complexity guarantees: We use the noisy identifiability guarantees to prove that for any consistent score difference estimator, we can guarantee ( ϵ, δ ) -PAC guarantees for CRL identifiability objectives.

Appendix E - RKHS-based score estimator: Finally, we use the finite sample guarantees provided by the RKHS-based score estimator of [12] to adapt the sample complexity statements for a generic estimator to explicit guarantees in terms of model-dependent constants.

To be used throughout the proof, we define the correlation matrices of d m Z as

We recall the correlation matrices R m X and ˆ R m X specified in (16). For a subset M⊆ [ n ] , we use the following shorthand notations for sums of the matrices in { R m X : m ∈ M} and { ˆ R m X : m ∈ M} .

B Identifiability guarantees in infinite sample regime

In this section, we provide the identifiability guarantees of Algorithms 1-3 with no error and probability 1 when using { R m X : m ∈ [ n ] } as input.

Denote the image of a function f : R n → R k for any k ∈ Z + by im ( f ) , that is,

First, we note a property of the column space of correlation matrices.

Lemma 6. For a continuous function f : R n → R k , span of the image of f equals to

Proof: Note that two subspaces are equal if and only if their orthogonal complements are equal. We denote the orthogonal complement of span im ( f ) by

The orthogonal complement of the column space of a symmetric matrix is the null space, which we denote by

Since any correlation matrix is positive semidefinite, using Proposition 2, (39) can be written as

Since f is a continuous function, so is y ⊤ · f for any y ∈ R k . We can then use [5, Proposition 2], which is stated below for the sake of conciseness.

Proposition 1 ([5, Proposition 2]) . Consider two continuous functions f, g : R n → R . Then, for any α > 0 ,

̸

Under this proposition, (41) can be written as

which is the same as the definition of S 1 in (38). This concludes the proof.

Corollary 1. For a continuous function f : R d → R k , the span of the image of f ◦ G equals to the column space of

We can specialize this lemma for d m Z and d m X for any m ∈ [ n ] , which yields

̸

and

B.1 Proof of Lemma 2

We can state (17) from Lemma 1 equivalently as

̸

Recall from (18) in Lemma 1 that d m X is related to d m Z by

Then, due to (47), for any given z ∈ R n and corresponding x = G · z , we have

Since d m X ( x ) lies in the subspace span { ( G † ) i, : : i ∈ pa( I m ) } for any x ∈ col ( G ) , we have

Due to (46), we have

which concludes the proof.

B.2 Encoder recovery

We can use Lemma 2 to show that in the infinite sample regime, Algorithm 3 works correctly. Specifically, let us show the following.

Lemma 7 (Infinite samples - Encoder) . For any v ∈ col ( R m X ) , we have

̸

Proof: Due to Lemma 2, for any v ∈ col ( R m X ) , we have

In other words, any v ∈ col ( R m X ) can be expressed as

for some constants { c i : i ∈ pa( I m ) } . Then, we have

Since G has full column rank, the pseudoinverse G † satisfies G † · G = I n , that is,

where e i denotes the i -th standard basis vector. Substituting this, (55) becomes

Note that i -th entry of a row vector w ∈ R n is given by w · e i . Therefore, we have

This concludes the proof.

B.3 Equivalence of function rank to correlation rank

In this section, we adapt the workhorse lemmas of [5] to our setting.

Lemma 8 ([5, Lemmas 5 and 6]) . Consider set A ⊆ [ n ] such that I A ≜ { I m : m ∈ A} is ancestrally closed, i.e., pa( I A ) = I A . Then,

Also, under Assumption 1, for any k ∈ A , we have

We first note that due to (18) and the definitions of R m X and R m Z in (16) and (34), we have

where G † is a full row rank matrix. Therefore, for any B ⊆ [ n ] ,

Next, we note that for a function f : R n → R k , the notation R ( f ) in [5] is used to denote the 'rank' of a function, and it corresponds to dimspanim ( f ) in the notation of this paper. Using Proposition 3 and Lemma 6, we can see that the notation R ( F ) for a set of functions F can be given as

Using (66) and (62), we can adopt [5, Lemmas 5 and 6] using { R m X : m ∈ [ n ] } matrices.

We note that (59) and (60) together with Lemma 2 imply that, for any ancestrally closed A ⊆ [ n ] and k ∈ A for which there exists j ∈ A \ { k } such that I k ∈ pa( I j ) , we have

We note that due to the property G † · G = I n , we have, for any M⊆ [ n ] ,

and

B.4 Causal order recovery

We will show that Algorithm 1 gives a valid causal order in the infinite sample regime using Lemma 8. Lemma 9 (Infinite samples - Causal order) . Setting η = 0 , I ◦ π is a valid causal order, where π is the output of Algorithm 1 using { R m X : m ∈ [ n ] } .

Proof: The proof of this lemma follows closely to the proof of [5, Lemma 7]. Note that setting η = 0 means that approximate rank and column/null spaces are actually equal to the standard rank and column/null space definitions. We note that I ◦ π is a valid causal order if and only if, for any t ∈ [ n ] , we have

We prove (70) by induction as follows.

In the base case, for t = n , we have V t = [ n ] . Therefore, using Lemma 8, we see that

̸

Therefore, for t = n , rank ( R V t \{ k } X ) = t -1 in Algorithm 1 if and only if k ∈ V t satisfies

Since { I π i : i > t } = ∅ for t = n , the base case is proved by choosing any such k as π t .

For the induction step at time t ∈ { n -1 , . . . , 1 } , we assume that, for any u ∈ { t +1 , . . . , n } , we have de( I π u ) ⊆ { I π i : i > u } . By construction, V t = [ n ] \ { π u : u > t } . Therefore, I V t is

ancestrally closed: For any k ∈ I V t , k cannot be a descendant of I [ n ] \V t = { I π u : u > t } due to the induction hypothesis. In other words, pa( I V t ) = I V t . Therefore, we can use Lemma 8 as

̸

Therefore, for any t ∈ { n -1 , . . . , 1 } , rank ( R V t \{ k } X ) = t -1 in Algorithm 1 if and only if k ∈ V t satisfies

Therefore, choosing any such k as π t satisfies the induction hypothesis for t as well, and the proof by induction is concluded.

B.5 Graph recovery

Lemma 10 (Infinite samples - Graph) . Setting η = γ = 0 , the collective output of Algorithms 1 and 2 using { R m X : m ∈ [ n ] } is equal to the graph isomorphism of the transitive closure of G by permutation I .

Proof: The proof of this lemma follows closely to the proof of [5, Lemma 8]. Note that setting η = 0 means that approximate ranks and column/null spaces are equal to the standard definitions. Similarly, γ = 0 means that approximate orthogonality is equivalent to the standard orthogonality definition. We first note that the transitive closure of two DAGs are equal if and only if their transitive reductions are equal. Therefore, the lemma statement is equivalent to

We will prove this result by induction over the transitive reduction edges. In the rest of the proof, using Lemma 9, we will leverage that I ◦ π is a valid causal order.

In the base case, we have t = n -1 , and thus necessarily j = n . The only possible path between I π t to I π j is therefore a transitive reduction edge. Since we did not add any children to node π t yet, the set M t,j is equal to V j \ { π t } . Noting that I V j is ancestrally closed and the only possible child of I π t is I π j when I ◦ π is a valid causal order, we can use Lemma 8 to obtain

Specifically, using (67), we can specify the exact column spaces we can obtain from M t,j as

Then, using (68), the null space of R M t,j X is given by

The test in Algorithm 2 involves checking whether the column space of R V t X and the null space of R M t,j X are orthogonal. Note that I V t is ancestrally closed, therefore, using (67), we get

Next, we note that the orthogonality test which uses orthonormal bases of subspaces (Definition 5) is equal to the following. Let A ∈ R d × k 1 and B ∈ R d × k 2 be full column rank matrices. Then,

This is equivalent to

This holds since A · A † is the orthogonal projection operator onto col ( A ) . Therefore, given orthonormal bases A 0 and B 0 for A and B , we have A · A † = A 0 · A ⊤ 0 and B · B † = B 0 · B ⊤ 0 . Then,

which is precisely the metric used in Definition 5. To use this property, let us first consider the matrix G : , A which is formed by stacking the independent column vectors in set { G : ,i : i ∈ A} for some subset A ⊆ [ n ] . Since G has full column rank, we see that the pseudoinverse of G : , A is exactly given by

where the matrix ( G † ) A , : is formed by stacking the independent row vectors in set { ( G † ) i, : : i ∈ A} . Using this notation, if we have

then the orthogonality metric is equal to

Due to the construction of sets V t and M t,j , we have I V t \ I M t,j = I π t . Therefore, the only entry in I V t that may be missing from pa( I M t,j ) is I π t . Therefore, previous equation is equal to

In summary, if (84) holds, the orthogonality inspected in Algorithm 2 happens if and only if

In the base case, since t = n -1 , this holds if and only if I π t ∈ pa tr ( I π j ) , and the base case is done.

For the induction hypothesis, we assume that for any u ∈ { n -1 , . . . , t + 1 } , all the transitive reductive edges has been correctly identified, that is, for any i ∈ [ n ] , π u ∈ ˆ pa tr ( π i ) if and only if I π u ∈ pa tr ( I π i ) . We show that the induction hypothesis is also satisfied for t and j ∈ [ n ] using induction on j .

In the base case, we have j = t +1 . This case works exactly the same way as the t = n -1 and j = n case. For the induction hypothesis, assume that for any u ∈ { n +1 , . . . , j -1 } , all the transitive reductive edges between I π t and I π u have been correctly identified, that is, π t ∈ ˆ pa tr ( π u ) if and only if I π t ∈ pa tr ( I π u ) . We prove that the hypothesis also holds for j as follows. First, we claim that I M t,j ∪{ π t } is an ancestrally closed set. Note that I V j is an ancestrally closed set by construction. Since the induction hypothesis ensures that all transitive reduction relations between I π u and I π i for any u ∈ { t +1 , . . . , n -1 } and i ∈ [ n ] are recovered, this also means that transitive closure relations, that is, descendants, are also recovered. Namely, the induction hypothesis implies

Next, the induction hypothesis for j implies that for any k ∈ { t + 1 , . . . , j -1 } , we have also identified all the transitive reductive edges I π t → I π k . Therefore, for set M t,j ∪{ π t } = V j \ ˆ ch( π t ) , all the elements in ˆ ch( π t ) has the property that their descendants have been fully identified, and therefore are excluded from the set ( ˆ G keeps the transitive closure), which means the set I M t,j ∪{ π t } is ancestrally closed. Since this set is ancestrally closed, we have

Proof: First, note that

Next, we apply the spectral norm.

Note that the first term is the spectral norm of a covariance matrix, and is upper bounded by

For the second term, we can upper bound the spectral norm by Frobenius norm as

Using Cauchy-Schwarz inequality,

In the base case t = n , we had shown that this was a sufficient condition for ensuring that Algorithm 2 detects the transitive reductive edges correctly. Therefore, we can detect any existing transitive reduction edges between I π t and I π j for any j and t , which concludes the proof.

C Identifiability guarantees under bounded score estimation noise

In this section, we show that as long as the error in the estimate of the score differences, max m ∈ [ n ] ∥ ˆ d m X -d m X ∥ p X , is upper bounded by a model-dependent constant, the results in the infinite data regime still hold.

The first key observation is that the error

Lemma 11.

∥

ˆ

R

m

X

-

R

m

X

∥

2

is bounded in terms of

∥

ˆ

d

m

X

-

d

m

X

∥

p

X

.

In high signal-to-noise ratio regime for score differences, that is, when

the error in correlation matrix R m X is bounded by the error in d m X as

Therefore, we can upper bound ∥ ˆ R m X -R m X ∥ 2 by

Finally, we note that if (95) holds, then this bound simplifies to (96).

Our finite sample CRL algorithms use approximate column and null spaces of (sums of) noisy score difference correlation matrices ˆ R M X , where M ⊆ [ n ] . Therefore, we need to ensure that these approximate subspaces (i) have the same rank as, and (ii) are close to their noise-free counterparts.

First, we show that under bounded score difference estimation noise, the approximate rank of ˆ R M X is equal to the rank of R M X . To show this, we will leverage Weyl's inequality.

Lemma 12 (Weyl's inequality) . For any two symmetric k × k real matrices A , B , we have

where λ ( A ) for symmetric matrix A denotes the vector of eigenvalues of A ordered in ascending order, and ∥ v ∥ ∞ is the maximum absolute value among the entries of v .

We can now show that the approximate rank of ˆ R m X matrices and sums are equal to the rank of R m X under proper threshold selection and low error.

Lemma 13. Define η ∗ as the minimum nonzero eigenvalue among { R M X : M⊆ [ n ] } , that is,

̸

(i) For η ∈ (0 , η ∗ ) , if

then, for any m ∈ [ n ] , we have

(ii) For η ∈ (0 , η ∗ ) , if

then, for any M⊆ [ n ] , we have

Proof: Let us investigate the approximate rank of ˆ R M X under threshold η for any M ⊆ [ n ] . If the eigenvalues of ˆ R M X that correspond to the eigenvalues of the null space of R M X are below the threshold η , and the eigenvalues of ˆ R M X that correspond to the eigenvalues of the column space of R M X are above η , then the approximate rank of ˆ R M X is equal to the rank of R M X . Let us denote the maximum amount any eigenvalue of ˆ R M X differs from the corresponding eigenvalue of R M X by

Thus, the eigenvalues of the noisy correlation matrix ˆ R M X that correspond to the null space eigenvalues of R M X is at most χ . Similarly, eigenvalues of the noisy correlation matrix ˆ R M X that correspond to the column space eigenvalues of R M X is at least

̸

̸

then using threshold η ensures that the approximate rank of ˆ R M X is equal to the rank of R M X . We can generalize this statement as follows. First, note that, by definition,

̸

Therefore, if

Then, the rank statement about ˆ R M X is correct if χ satisfies

This statement can be equivalently written as

Next, we find an upper bound on χ in terms of ∥ R m X -R m X ∥ 2 using Weyl's inequality (Lemma 12).

Therefore, for any η ∈ (0 , η ∗ ) , leveraging the condition (120), we have

(i) If |M| = 1 and M = { m } , we can use (123) to show that the condition

ensures that

(ii) For any generic M , we can use (124) to show that the condition

ensures that

This concludes the proof.

Since we have a sufficient condition to ensure that approximate ranks of { ˆ R M X : M⊆ [ n ] } are equal to the ranks of { R M X : M⊆ [ n ] } , now we can state a sufficient condition for finding a valid causal order.

Lemma 14 (Bounded noise - Causal order) . Let η ∈ (0 , η ∗ ) . If

then I ◦ π is a valid causal order, where π is the output of Algorithm 1.

Proof: We note that Algorithm 1 only uses approximate rank statements for { ˆ R M X : M⊆ [ n ] } . Therefore, if the approximate rank statements are equal to the ranks of { R M X : M⊆ [ n ] } , then, the infinite sample guarantees of Algorithm 1 provided in Lemma 9 immediately transfer to the noisy R m X regime.

Next, we note that Algorithms 2 and 3 use the direction information of the approximate column and null spaces of { ˆ R M X : M⊆ [ n ] } as well. Therefore, we need to bound the 'distance' of the estimated subspaces from their noise-free counterparts. The main results we will use for this purpose are the Davis-Kahan sin θ theorems [19], specialized for approximate column and null spaces.

Lemma 15 ([19, sin θ theorem]) . Consider two k × k real symmetric matrices A and ˆ A . Denote the orthonormal bases of the column and null spaces of A by A 1 and A 0 , respectively, such that

where C 1 is a symmetric matrix with the same eigenvalues as A . Similarly, denote the orthonormal bases for the approximate column and null space of ˆ A by ˆ A 1 and ˆ A 0 , such that

where eigenvalues of ˆ C 0 are that of the approximate null space of ˆ A , and the eigenvalues of ˆ C 1 are that of the approximate column space of ˆ A . Define

where λ max and λ min denote the maximum and minimum eigenvalue of a symmetric matrix. Then, given that the approximate rank of ˆ A is equal to the rank of A and χ > 0 , we have

and

Lemma 16 ([19, Symmetric sin θ theorem]) . Under the setting described in the sin θ theorem above, we have and

We note that χ in these sin θ theorems can be bounded using Weyl's inequality (Lemma 12) and the minimum nonzero eigenvalue of matrix A as follows.

Therefore,

Using R M X for A and ˆ R M X for ˆ A , and using the definition of η ∗ in (110), we obtain

In order to ensure that the approximate rank of ˆ R M X is equal to the rank of R M X , it suffices that ∥ ˆ R M X -R M X ∥ 2 satisfies due to (121). If this holds, we get

Therefore, the left-hand sides of (133)-(136) are upper bounded by

We can use Davis-Kahan symmetric sin θ theorem (Lemma 16) to prove that encoder estimation error is upper bounded by the error in ˆ R m X .

Lemma 17 (Bounded noise - Encoder) . Let η ∈ (0 , η ∗ ) . If (125) holds, then the output H of Algorithm 3 satisfies

where for all i ̸∈ pa( j ) , C pa satisfies ( C pa ) i,j = 0 , and

Proof: We prove this result by considering each row m of H separately. In Algorithm 3, we select H m, : from col ( ˆ R m X ; η ) for each m ∈ [ n ] . We start by noting that in Lemma 7, we have shown that using the noise-free score difference correlation matrices, we get

̸

We can decompose any v ∈ col ( ˆ R m X ; η ) as

where A 0 and A 1 are the orthonormal bases of the null and column spaces of R m X in line with the notation of the Davis-Kahan sin θ theorem. Note that A 1 · A ⊤ 1 is the orthogonal projection operator onto the column space of R m X , therefore, using (145), we have

̸

Therefore, the 'erroneous' entries in v ⊤ · G can be written as

Note that if we choose v ∈ col ( ˆ R m X ; η ) as row m of H , we have

Since v ∈ col ( ˆ R m X ; η ) , we have ˆ A 1 · ˆ A ⊤ 1 · v = v , where ˆ A 1 is the orthonormal basis of the approximate column space of ˆ R m X . Using this and ∥ v ∥ 2 = 1 , we can write (149) as

Using Davis-Kahan sin θ theorem (Lemma 15), when (125) holds, we can further upper bound the m -th row of the error term by

Note that the spectral norm is upper bounded by the Frobenius norm. Therefore,

Next, we investigate the approximate orthogonality test used for Algorithm 2 under noisy score difference correlation matrices.

Lemma 18. Let A , B , ˆ A , ˆ B be k × k symmetric matrices, such that

where [ A 0 , A 1 ] , [ B 0 , B 1 ] , [ ˆ A 0 , ˆ A 1 ] , [ ˆ B 0 , ˆ B 1 ] are all k × k orthogonal matrices, and shapes of A 0 -ˆ A 0 and B 0 -ˆ B 0 match. Given that the spectra of (i) C 0 and ˆ C 1 , (ii) C 1 and ˆ C 0 , (iii) D 0 and ˆ D 1 , and (iv) D 1 and ˆ D 0 are all separated with gap χ , we have

(i) If ∥ A ⊤ 0 · B 0 ∥ 2 = 0 , that is, if col ( A 0 ) ⊥ col ( B 0 ) , then,

(ii) If ∥ A ⊤ 0 · B 0 ∥ 2 = µ = 0 , that is, col ( A 0 ) ̸⊥ col ( B 0 ) , then,

̸

Proof: We first prove a perturbation bound on the product of orthogonal projectors.

Using Davis-Kahan symmetric sin θ theorem (Lemma 16) yields

We use this result in two separate cases: col ( A 0 ) ⊥ col ( B 0 ) and col ( A 0 ) ̸⊥ col ( B 0 ) .

(i) Case 1: col ( A 0 ) ⊥ col ( B 0 ) . In this case, (164) immediately yields (158).

̸

(ii) Case 2: col ( A 0 ) ̸⊥ col ( B 0 ) . Let us define ∥ A ⊤ 0 · B 0 ∥ 2 = µ = 0 . We can use (164) and triangle inequality to get

Rearranging the terms yields (159).

Finally, we show that the graph estimation is correct under bounded noise.

Lemma 19 (Bounded noise - Graph) . Let η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) , where

Under Assumption 1, the output ˆ G of Algorithm 2 is the graph isomorphism of the transitive closure of the true latent graph G if

Proof: Algorithm 2 uses approximate column and null spaces of R M X for different M ⊆ [ n ] . Therefore, the generic sufficient condition (113), that is,

ensures that the approximate ranks of all ˆ R M X matrices investigated in the algorithm will be equal to the rank of R M X . To ensure that the estimated graph will be correct, we must guarantee that the approximate orthogonality tests will also give correct results. Using Lemma 18, we can show that the test metric is upper bounded in terms of max m ∈ [ n ] ∥ ˆ R m X -R m X ∥ 2 . We do so by setting A = R V t X , ˆ A = ˆ R V t X , B = R M t,j X , ˆ B = ˆ R M t,j X ; and A 0 as the orthonormal basis of the column space of A , and B 0 as the null space of B . As shown in (141), (113) ensures that the spectral gap χ in Lemma 18 statement is at least χ > η ∗ . In Lemma 10, we show that in the infinite sample, noise-free regime, when investigating ( t, j ) , the test metric is given by

Since the approximate orthogonality test needs to distinguish even the weakest instance of nonorthogonality from orthogonality, we define

Using Lemma 18, we see that for ( t, j ) for which I t → I j exists in G , if (113) holds, then

Similarly, for ( t, j ) for which I t → I j does not exist in G , if (113) holds, then

Therefore, for a choice of γ ∈ (0 , γ ∗ ) to yield correct orthogonality decisions, it suffices that

or, equivalently,

In other words, when using γ ∈ (0 , γ ∗ ) as the approximate orthogonality threshold, we can correctly identify orthogonal subspaces from non-orthogonal ones when

Due to Lemma 10, if all the approximate rank and orthogonality tests are correct, then the output ˆ G of Algorithm 2 is the graph isomorphism of the transitive closure of the true latent graph G .

D Sample complexity for generic consistent score estimator

In this section, we will transform the bounded noise identifiability guarantees in Appendix C to sample complexity statements about a generic consistent score difference estimator. Specifically, recall the consistent score difference estimator model from (15).

Recall the definitions of the constants in (24),

ˆ m

First, we derive the sufficient samples for ( ϵ, δ ) -approximating R X using Lemma 11. Lemma 20. For any ϵ > 0 and δ > 0 , using N R ( ϵ, δ ) samples suffice to ensure that

where

Proof: In Lemma 11, in order to bound the error in ˆ R m X for a specific m ∈ [ n ] , we require

To ensure this for all m ∈ [ n ] , it suffices that

Under this condition, Lemma 11 shows that the error in correlation matrices R m X is bounded by the error in d m X as

To get an m -agnostic bound, we can take max of both sides to get

In other words, to achieve ϵ error in estimating ˆ R m X , the error in estimating d m X suffices to be upper bounded by

In summary, (184) and (187) together gives that

suffices to ensure

Using the sample complexity of ˆ d m X , this means that

samples suffice to ensure this error upper bound on ˆ R m X with probability at least 1 -δ . Next, we restate and prove the sample complexity of estimating the causal order in Lemma 4.

Lemma 4 (Sample complexity - Causal order) . Let η ∈ (0 , η ∗ ) . Under Assumption 1, for any δ > 0 , N rank ( δ ) samples suffice to ensure that with probability at least 1 -δ , I ◦ π is a valid causal order, where π is the output of Algorithm 1, where

Proof: Lemma 14 states that as long as ˆ R m X error is upper bounded by η ∗ n , the output of Algorithm 1 is correct. Then, using Lemma 20, we see that

samples suffice to ensure that the estimated causal order is correct with probability at least 1 -δ . Next, we prove the sample complexity of estimating the latent graph.

Theorem 1 (Sample complexity - Graph) . Let η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) . Under Assumption 1, for any δ > 0 , N G ( δ ) samples suffice to ensure that collective output ˆ G ( X N ) of Algorithms 1 and 2 satisfies δ -PAC graph recovery, where

Proof: Lemma 19 states that the estimated graph is correct if

Therefore, Lemma 20 implies that

samples suffice for correct graph recovery with probability at least 1 -δ . Similarly, we prove the sample complexity of estimating the causal variables.

Theorem 2 (Sample complexity - Variables) . Let η ∈ (0 , η ∗ ) . For any ϵ > 0 and δ > 0 , N Z ( ϵ, δ ) samples suffice to ensure that the output H ( X N ) of Algorithm 3 satisfies ( ϵ, δ ) -PAC causal variables recovery, where

Proof: Lemma 17 states that if the input error is bounded by

then the estimated encoder achieves error rate ϵ

Therefore, to ensure ( ϵ, δ ) -PAC causal variables recovery, it suffices that, with probability at least 1 -δ , the input error on ˆ R m X is upper bounded by

Using Lemma 20, this implies that

samples suffice for ( ϵ, δ ) -PAC causal variables recovery.

E Sample complexity for RKHS-based score estimator

In this section, we adopt the RKHS-based score estimator of [12], state its assumptions, derive its noise model for score difference estimation, and use this model to make the sample complexity upper bounds derived in Appendix D explicit.

The score estimator in [12] requires the support of the distribution to be an open subset of its domain [12, Assumption B.1]. Therefore, we estimate the score function of X through its orthogonal projection to the n dimensional support manifold col ( G ) . The projection matrix can be constructed from observed data samples with probability 1 as long as N ≥ n .

Next, we state Lemma 5 formally including all the assumptions required.

Lemma 5 (Formal) . Denote a trace-class matrix-valued kernel by K : R d × R d → R d × d , denote its associated RKHS by H K , and define κ ≜ sup x ∈ col ( G ) tr K ( x, x ) . Define an integral operator on H K as

where p is a pdf. Assume that there exists f 0 ∈ H K and f m ∈ H K for all m ∈ [ n ] such that s X = L K ,p X f 0 and s m X = L K ,p m X f m . Then, using Tikhonov regularization, under [12, Assumptions B.1B.5], for any m ∈ [ n ] , for any δ ∈ (0 , 1) and N ≥ (2 √ 2 κ 2 log 8 n/δ ) 4 , with probability at least 1 -δ , score estimator in [12, Theorem B.1] satisfies

where C is a sample independent constant that depends only on p X , p m X for m ∈ [ n ] and the structure of H K .

Proof: We use the fact that for a function f ∈ H K , we have

Therefore, even if we estimate s m X using data from p m X but evaluate it on data from p X , we can still bound the MSE of the estimate using the Hilbert norm. The result states that the Hilbert norm error bound is, for any m ∈ { 0 } ∪ [ n ] , and if N ≥ (2 √ 2 κ 2 log 4 /δ ) 4 ,

Since ∥ · ∥ p X is a norm, we have

Combining with the Hilbert norm error bound, taking union bound and defining C ≜ 2 C 1 ∥ L K ,p X ∥ op , we get

Note that the noise model in (15) requires a bound on max m ∈ [ n ] ∥ ˆ d m X -d m X ∥ p X . By taking a union bound, we get

By replacing δ with δ/ 2 n , we get, when N ≥ (2 √ 2 κ 2 log 8 n/δ ) 4 ,

which concludes the proof.

Corollary 2. The error bound given in this result can be immediately transformed into a sample complexity statement. Specifically,

where

Using this N ( · , · ) function, we can immediately make the sample complexity results in the previous section explicit. The theorem statements are also stated here for completeness.

Theorem 3 (RKHS-based sample complexity - Graph) . Let η ∈ (0 , η ∗ ) and γ ∈ (0 , γ ∗ ) . Under Assumption 1 and the conditions of Lemma 5, N G ( δ ) samples sufficient to ensure that the collective output ˆ G ( X N ) of Algorithms 1 and 2 satisfies δ -PAC graph recovery, where

Theorem 4 (RKHS-based sample complexity - Variables) . Let η ∈ (0 , η ∗ ) . Under the conditions of Lemma 5, N Z ( ϵ, δ ) samples ensure that the output H ( X N ) of Algorithm 3 satisfies ( ϵ, δ ) -PAC causal variables recovery, where

and

F Properties of positive semidefinite matrices

In this section, we present properties of positive semidefinite (PSD) matrices that are commonly used in our proofs.

Proposition 2. For a real k × k PSD matrix A , the null space of A can be specified using the quadratic form as

̸

̸

Proof: For a vector v ∈ R k , Av = 0 implies v ⊤ Av = 0 unconditionally. We prove the reverse direction by showing that Av = 0 implies v ⊤ Av = 0 . First, since A is PSD, it has a Cholesky decomposition A = U ⊤ U for some matrix U ∈ R k × k . If Av = U ⊤ Uv = 0 , then Uv = 0 . This implies v ⊤ Av = ∥ Uv ∥ 2 2 = 0 . This concludes the proof.

̸

̸

̸

Proposition 3. If A , B are real k × k PSD matrices, then

where + denotes the Minkowski sum between sets, which is defined, for any two subsets A and B of a vector space, as

Proof: Using Proposition 2, we can write null ( A + B ) as

Since A and B are both PSD, for any v ∈ R k , we have v ⊤ Av ≥ 0 and v ⊤ Bv ≥ 0 . Therefore, v ⊤ ( A + B ) v = 0 holds if and only if v ⊤ Av = 0 and v ⊤ Bv = 0 . That is,

We note that, due to Proposition 2, the right hand side of this expression is equal to null ( A ) ∩ null ( B ) , which concludes the proof for (215).

Next, note that the column space of a symmetric matrix is the orthogonal complement of its null space. Then, using (215), we have

Since null ( A ) and null ( B ) are subspaces of a finite dimensional vector space, we have

This concludes the proof for (216).

G Hyperparameter estimation

In this section, we provide minor modifications to our algorithms that enable constructing rough estimates for threshold upper bounds η ∗ and γ ∗ in practice.

Estimating η ∗ : We note that in Algorithm 1, at step t , all matrices investigated must have either rank t or t -1 . Therefore, the t -1 -th eigenvalues of all of these matrices are non-zero, or equivalently we have ≥ η ∗ . This observation lets us to iteratively build and refine an upper bound on η ∗ during Algorithm 1, which we can use as a surrogate of η ∗ in the subsequent Algorithm 2 and 3. For Algorithm 1 itself, at time t , we can pick k ∈ V t with minimal t -th eigenvalue-this is a threshold-free way of estimating the causal order.

Estimating γ ∗ : We first note that definition in (23) depends entirely on the ground truth decoder matrix G and its pseudoinverse. Secondly, we note that Algorithm 3 can be used independently of Algorithm 2 to generate an estimate H of G † up to some estimation error and possible mixing among rows. We can compute the value

which would be equal to γ ∗ had H = G † , and use it as our estimate of γ ∗ .

and

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Sections 4 and 5 establish the results claimed in the abstract and introduction. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work, e.g., linear transformations, are clarified throughout the paper and discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions for the theoretical results are clearly given in the theorem statements, and the proofs of the results are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The algorithm details are described in Section 4. The experimental procedure is described in Section 6. The codebase for the experiments can be found at https: //github.com/acarturk-e/finite-sample-linear-crl .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codebase for the experiments can be found at https://github.com/ acarturk-e/finite-sample-linear-crl .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental procedure is described in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The only experimental result with error bars is Figure 1a, in which the 95% error bars are reported but are indistinguishable from the plot lines.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Experiments are run on a single commercial laptop CPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and confirm that the research in this paper conforms with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is mostly theoretical and does not pose potential negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is mostly theoretical and does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Portions of the publicly available code of [5], available under Apache 2.0 license, is adopted in the code of our experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The codebase for the experiments can be found at https://github.com/ acarturk-e/finite-sample-linear-crl , and is released under Apache 2.0 license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInterventional causal representation learning aims to recover the causal graph for latent variables and simultaneously recover the latent variables. While identifiability has been established for the infinite sample regime, this is not practical. This paper aims to provide PAC-style bounds on identifiability (of both the graph and latent variables) in the finite-sample regime, assuming stochastic soft interventions and linear transformations from the latent to the observation space. Three prototypical algorithms (causal order estimation, graph estimation, and inverse transformation estimation) are proposed to estimate the graph and the latent variables, which pave the foundation, from which  PAC-style bounds have been established. Numerical assessments are conducted to provide complementary insights into the sample complexity results.\n\nSTRENGTHS:\nIdentifiability analysis is often conducted in the infinite sample regime, limiting its practical applicability. This paper extends identifiability analysis to the finite sample regime, representing a significant step forward.\n\nWEAKNESSES:\nThe bounds are rather loose, which limits its practical applicability. Apart from that, these bounds do not provide sufficient guidance on designing better algorithms recovering latent variables and graphs. Having said so, I still welcome this type of theoretical papers.\n\nQUESTIONS:\nI wonder if causal order estimation is necessary. Since the latent variables have yet to be estimated, their values and semantics are flexible. This implies that any given causal order could be acceptable, as these variables can be learned to fit the specified orders. Some edges would be pruned later, which can be done in the graph estimation stage.\n\nAdditionally, I wonder if there are other quantities beyond score differences and the correlation of score differences that could be used to recover the graph and the latent variables.\n\nFurthermore, it would be beneficial if the analysis could assess which variables and which parts of the graph are more prone to error, and to what degree, providing feedback on the estimated/recovered variables and graphs.\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper establishes finite sample guarantees for recovering the latent causal graph under stochastic soft interventions and observations under a unknown linear transformation. Experiments are conducted and code is given.

STRENGTHS:
The paper is well motivated and the contributions are clear.

WEAKNESSES:
I did not check the proofs in detail but I do not spot any glaring weaknesses.

The experiments are rather small scale (up to 10 nodes).

QUESTIONS:
- What is $P_I$ in equation (12)?
- In practice, do we need to know $\eta^*$ or $\gamma^*$ to invoke your results? How do we know enough samples have been collected?

LIMITATIONS:
Nil
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides finite-sample identifiability results for recovering the latent causal graph and the generating latent variables given observations in a high-dimensional space that have been generated by the latent variables by a linear transformation and single-node soft interventional data with interventions on all nodes. Previous work provides identifiability results only in the infinite-sample regime. The algorithms rely on score-based estimators which have been studied in literature. Specifically the correlation matrix of the score differences are shown to be constrained by the graph and the linear transformation matrix. To recover the graph, first a topological order on the variables is estimated. Verifying whether a particular edge exists is checked by a necessary and sufficient condition on the approximate orthogonality of the column space and null space. For the variables recovery, it is sufficient to consider a unit vector in the column space. An existing RKHS-based score estimator is used to obtain the final sample-complexity results.\n\nSTRENGTHS:\nCausal representation learning is an active field of study and theoretical work that provides finite-sample identifiability results is quite relevant in practice and of interest to the community. Overall the paper is well-written with a clear flow of ideas that makes it easy to follow. \n\nI liked that the proposed method has a decoupling between the score estimators and the rest of the algorithms with proofs for the infinite-sample regime separate. This helps us to understand that the difference in quality of finite-sample data versus infinite sample data is the score-estimators.\n\nWEAKNESSES:\nA few weaknesses (with the major one being point 1) follow:\n\n1. A major confusion for me was that the setting of the thresholds was not clear. If the true scores are not known, then how is \\eta set from \\eta* which depends on the unknown true score estimates. Similarly for \\gamma that depends on \\gamma^* that depends on the true transformation matrix. What am I missing? \n\n2. Overall, the paper has multiple instances of notations not defined which make the paper difficult to parse. For example, Definition 2 has P_I, C_{pa}'s diagonal entries undefined? Lemma 2 has the pseudoinverse subscript undefined. \n\n3. I also think that the experimental section needs basic details. One suggestion would be to cut out the final RKHS sample complexity bounds that are just plugging (29) into Theorem 1 and 2. \n\n4. The proofs in the paper first derive results for the infinite-sample regime where score estimates are error-free. Finite-sample guarantees are then obtained by bounding the error in the score estimates. It would make the paper stronger if the infinite-sample regime results are compared with existing work. \n\n5. In Figure 1b, why are there spikes in the recovery rate even for worse score estimate MSE? Does this go against the main message of the paper that worse score estimates should imply poorer graph recovery and latent variable estimation? \n\n6. What is the reason behind defining the variables recovery as it is defined in Definition 2. Is there a previous instance of such a definition because this doesn't seem standard. Any rationale would help the reader.\n\nOverall, I think the paper is a solid contribution if these questions are clarified. But my current score can't improve given these questions.\n\nQUESTIONS:\nPlease refer to the weakness section for major issues. A few miscellanous points:\n1) I haven't checked the details but it struck me as weird to notice that there is a specific value of \\gamma in (20) that verifies whether an edge exists or not. Is there any intuition for this? \n2) Typos - (22) M \\subseteq [n], Line 179. \n3) In Figure 1b, is there a reason why even for better score estimates, there is no point for the 10 nodes case?\n\nLIMITATIONS:\nImplicitly addressed in the conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper provides finite sample analysis for causal representation learning with general latent causal models, single-node soft interventions, and linear mixing functions. Sample complexity for identifying graphs up to ancestors and identifying latent variables up to mixing with parent variables have been studied.

STRENGTHS:
1. To the best of my knowledge, this is the first finite sample analysis for interventional CRL, which is a very interesting problem with high practical value.

2. All theoretical results including definitions are organized clearly.

WEAKNESSES:
1. The considered CRL setting is a little bit restrictive compared to the recent ones. Specifically, the assumption of linear mixing function may restrict its usage in a lot of real-world scenarios. Meanwhile, the indeterminacy is non-trivial, and further process/disentanglement might still be needed. However, it is understandable since sample complexity might be much more difficult to get for more general models.

2. Perhaps it would be better if there was more discussion on the assumptions.

QUESTIONS:
1. How to make sure that all the imperfections in the experiments are due to score estimates?

2. Are there any intuitive ideas to extend the range of the finite sample analysis to more general settings in CRL?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13