Title: Consistency of Neural Causal Partial Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causal inference

Generative models

Partial identification
Abstract: Recent progress in Neural Causal Models (NCMs) showcased how identification and partial identification of causal effects can be automatically carried out via training of neural generative models that respect the constraints encoded in a given causal graph [Xia et al. 2022, Balazadeh et al. 2022]. However, formal consistency of these methods has only been proven for the case of discrete variables or only for linear causal models. In this work, we prove the consistency of partial identification via NCMs in a general setting with both continuous and categorical variables. Further, our results highlight the impact of the design of the underlying neural network architecture in terms of depth and connectivity as well as the importance of applying Lipschitz regularization in the training phase. In particular, we provide a counterexample showing that without Lipschitz regularization this method may not be asymptotically consistent. Our results are enabled by new results on the approximability of Structural Causal Models (SCMs) via neural generative models, together with an analysis of the sample complexity of the resulting architectures and how that translates into an error in the constrained optimization problem that defines the partial identification bounds.

Consistency of Neural Causal Partial Identification

Abstract

Recent progress in Neural Causal Models (NCMs) showcased how identification and partial identification of causal effects can be automatically carried out via training of neural generative models that respect the constraints encoded in a given causal graph [52, 3]. However, formal consistency of these methods has only been proven for the case of discrete variables or only for linear causal models. In this work, we prove the consistency of partial identification via NCMs in a general setting with both continuous and categorical variables. Further, our results highlight the impact of the design of the underlying neural network architecture in terms of depth and connectivity as well as the importance of applying Lipschitz regularization in the training phase. In particular, we provide a counterexample showing that without Lipschitz regularization this method may not be asymptotically consistent. Our results are enabled by new results on the approximability of Structural Causal Models (SCMs) via neural generative models, together with an analysis of the sample complexity of the resulting architectures and how that translates into an error in the constrained optimization problem that defines the partial identification bounds.

1 Introduction

Identifying causal quantities from observational data is an important problem in causal inference which has wide applications in economics [1], social science [20], health care [34, 18], and recommendation systems [9]. One common approach is to transform causal quantities into statistical quantities using the ID algorithm [49] and deploy general purpose methods to estimate the statistical quantity. However, in the presence of unobserved confounding, typically the causal quantity of interest will not be point-identified by observational data, unless special mechanisms are present in the data generating process (e.g. instruments, unconfounded mediators, proxy controls). In the presence of unobserved confounding, the ID algorithm will typically fail to return a statistical quantity and declare the causal quantity as non-identifiable.

One remedy to this problem, which we focus on in this paper, is partial identification, which aims to give informative bounds for causal quantities based on the available data. At a high level, partial identification bounds can be defined as follows: find the maximum and the minimum value that a target causal quantity can take, among all Structural Causal Models (SCMs) that give rise to the same observed data distribution and respect the given causal graph (as well as any other structural

constraints that one is willing to impose). Note that in the presence of unobserved confounding, there will typically exist many structural mechanisms that could give rise to the same observational distribution but have vastly different counterfactual distributions. Hence, partial identification can be formulated as solving a max and a min optimization problem [3]

where θ ( M ) is the causal quantity of interest, M ∗ is the true model, V is the set of observed nodes, P M ( V ) is the distribution of V in SCM M , C is a collection of causal models and G M is the causal graph of M (see Section 2 for formal definitions). Two recent lines of work explore the optimization approach to partial identification. The first line deals with discrete Structure Causal Models (SCMs), where all observed variables are finitely supported. In this case, (1) becomes a Linear Programming (LP) or polynomial programming problem and tight bounds can be obtained [37, 4, 5, 27, 44, 45, 40, 8, 54, 56, 14]. The second line of work focuses on continuous models and explores ways of solving (1) in continuous settings using various techniques [24, 32, 29, 38, 3, 41].

Recently, Xia et al. [51] formalized the connection between SCMs and generative models (see also [33] for an earlier version of a special case of this connection). This work showcased that SCMs can be interpreted as neural generative models, namely Neural Causal Models (NCMs), that follow a particular architecture that respects the constraints encoded in a given causal graph. Hence, counterfactual quantities of SCMs can be learned by optimizing over the parameters of the underlying generative models. However, there could be multiple models that lead to the same observed data distribution, albeit have different counterfactual distributions. Xia et al. [51] first analyze the approximation power of NCMs for discrete SCMs and employ the max/min approach to verify the identifiability of causal quantities, without the need to employ the ID algorithm. Balazadeh et al. [3] and Hui et al. [30], extend the method in [51] and re-purpose it to perform partial identification by solving the min and max problem in the partial identification formulation over neural causal models.

However, for SCMs with general random variables and functional relationships, the approximation error and consistency of this optimization-based approach to partial identification via NCMs has not been established. In particular, two problems remain open. First, given an arbitrary SCM, it is not yet known if we can find an NCM that produces approximately the same intervention distribution as the original one. Although Xia et al.[51] show it is possible to represent any discrete SCM by an NCM, their construction highly relies on the discrete assumption and cannot be directly generalized to the general case. Moreover, Xia et al. [51] use step functions as the activation function in their construction, which may create difficulties in the training process since step functions are discontinuous with a zero gradient almost everywhere.

Second, since we only have access to n samples from the true distribution, P M ( V ) , we need to replace the constraints in (1) with their empirical version that uses the empirical distribution of samples P M ∗ n ( V ) in place of the population distribution and looks for NCMs, whose implied distribution lies within a small distance from the empirical distribution. Moreover, even the NCM distribution is typically only accessible through sampling, hence we will need to generate m n samples from the NCM and use the empirical distribution of the m n samples from the NCM in place of the true distribution implied by the NCM. Thus, in practice, we will use a constraint of the form d ( P M ∗ n ( V ) , P M m n ( V )) ⩽ α n , where d is some notion of distribution distance and α n accounts for the sampling error. It is not clear that this approach is consistent, converges to the correct partial identification bounds, when the sample size n grows to infinity. Balazadeh et al. [3] only show the consistency of this approach for linear SCMs. Consistency results concerning more general SCMs is still lacking in the neural causal literature.

In this paper, we establish representation and consistency results for general SCMs. Our contributions are summarized as follows.

We show that under suitable regularity assumptions, given any Lipschitz SCM, we can approximate it using an NCM such that the Wasserstein distance between any interventional distribution of the NCM and the original SCM is small. Each random variable of the SCM is allowed to be continuous or categorical. We specify two architectures of the Neural Networks (NNs) that can be trained using common gradient-based optimization algorithms (Theorem 2, Theorem 3 and Corollary 1).

To construct the NCM approximation, we develop a novel representation theorem of probability measures (Proposition 1) that may be of independent interest. Proposition 1 implies that under certain assumptions, probability distributions supported on the unit cube can be simulated by pushing forward a multivariate uniform distribution.

We discover the importance of Lipschitz regularization by constructing a counterexample where the neural causal approach is not consistent without regularization (Proposition 2).

Using Lipschitz regularization, we prove the consistency of the neural causal approach (Theorem 4).

Related Work There exists a rich literature on partial identification of average treatment effects (ATE) [37, 4, 5, 27, 44, 45, 40, 8, 54, 56, 14, 7, 39, 26]. Balke and Pearl [4, 5] first give an algorithm to calculate bounds on the ATE in the Instrumental Variable (IV) setting. They show that regardless of the exact distribution of the latent variables, it is sufficient to consider discrete latent variables as long as all endogenous variables are finitely supported. Moreover, they discover that (1) is an LP problem with a closed-form solution. This LP-based technique was generalized to several special classes of SCMs [44, 8, 27, 46]. For general discrete SCMs, [14, 56, 55] consider transforming the problem (1) into a polynomial programming problem. Xia et al. [51] discover the connection between generative models and SCMs. They show that NCMs are expressive enough to approximate discrete SCMs. By setting C in problem (1) to be the collection of NCMs, they apply NCMs for identification and estimation.

For causal models with continuous random variables, the constraints in (1) become integral equations, which makes the problem more difficult. One approach is to discretize the constraints. [24] uses stochastic process representation of the causal model in the continuous IV setting and transforms the problem into a semi-infinite program. [32] relaxes the constraints to finite moment equations and solves the problem by the Augmented Lagrangian Method (ALM). The other approach is to use generative models to approximate the observational distribution and use some metric to measure the distance between distributions. [33] first propose to use GAN to generate images. Later, [29] uses Wasserstein distance in the constraint and transforms the optimization problem into a min and max problem. Similarly, [3] solves the optimization problem using Sinkhorn distance to avoid instability during training. They propose to estimate the Average Treatment Derivative (ATD) and use ATD to obtain a bound on the ATE. They also prove the consistency of the proposed estimator for linear SCMs. [41] uses a linear combination of basis functions to approximate response functions. [26] uses sieve to solve the resulting optimization problem in the IV setting. [19] proposes a neural network framework for sensitivity analysis under unobserved confounding.

Organization of this paper In Section 2, we introduce the notations and some basic concepts used throughout the paper. Next, in Section 3, we demonstrate how to construct an NCM so that they can approximate a given SCM arbitrarily well. Two kinds of architecture are given along with an approximation error analysis. In Section 4, we highlight the importance of Lipschitz regularization by giving a counterexample that is not consistent. Then, leveraging the previous approximation results, we are able to prove the consistency of this approach under regularization. Finally, we compare our method with the traditional polynomial programming method empirically in Section 4.1.

2 Preliminary

First, we introduce the definition of an SCM. Throughout the paper, we use bold symbols to represent sets of random variables.

Definition 1. (Structural Causal Model) A Structural Causal Model (SCM) is a tuple M = ( V , U , F, P ( U ) , G 0 ) , where V = { V i } n V i =1 is the set of observed variables; U = { U j } n U j =1 is the set of latent variables; P ( U ) is the distribution of latent variables; G 0 is an acyclic directed graph whose nodes are V . The values of each observed variable V i are generated by

where F = ( f 1 , · · · , f n V ) , Pa ( V ) is the set of parents of V in graph G 0 and U V i is the set of latent variables that affect V i . V i takes either continuous values R d i or categorical in [ n i ] . We extend graph G 0 by adding bi-directed arrows between any V i , V j ∈ G 0 if there exists a correlated latent variable pair ( U k , U l ) , U k ∈ U V i , U l ∈ U V j . We call the extended graph G M the causal graph of M .

When we write Pa ( V i ) , we refer to the parents of V i in the G 0 . To connect with the literature, the causal graph we define is a kind of Acyclic Directed Mixed Graph (ADMG), which is often used to represent SCMs with unobserved variables [49]. Note that we allow one latent variable to enter several nodes, which differs from the common definition. We use n U and n V to denote the number of latent variables and observable variables. Let T ⊂ V be a set of treatment variables. The goal is to estimate causal quantities under a given intervention T = t . Formally, the structural equations of the intervened model are

We denote V i ( t ) to be the value of V i under the intervention T = t and P M ( V ( t )) to be the distribution of V ( t ) in M . The notion of a C 2 component [51] is defined as follows.

Definition 2 ( C 2 -Component) . For a causal graph G , a subset C ⊂ V is C 2 -component if each pair V i , V j ∈ C is connected by a bi-directed arrow in G and C is maximal.

We provide a concrete example in Appendix A to explain all these notions. We will make the following standard assumption about the independence of latent variables. Note that since we allow latent variables to enter in multiple structural equations, this is more a notational convention and not an actual assumption. Also note that under this convention a bi-directed arrow essentially represents the existence of a common latent parental variable.

Assumption 1. All the latent variables in U are independent.

To deal with categorical variables, we assume that latent variables that influence categorical variables contain two parts: the shared confounding that influences the propensity functions and the independent noise that generates categorical distributions.

Assumption 2. The set of latent variables consists of two parts U = { U 1 , · · · , U n U } ∪ { G V i : V i is categorical } . Precisely, if V i ∈ V is a categorical variable, the data generation process of V i satisfies V i = arg max k ∈ [ n i ] { g V i k +log( f i ( Pa ( V i ) , U V i )) k } ∼ Categorical ( f i / ∥ f i ∥ 1 ) , where G V i = ( g V i 1 , · · · , g V i n i ) are i.i.d. standard Gumbel variables, U V i ⊂ { U 1 , · · · , U n U } .

This convention is without loss of generality at this point, but will be useful when introducing Lispchitz restrictions on the structural equation functions. The Gumbel variables in the assumption can be replaced by any random variables that can generate categorical variables. It can be proven that all discrete SCMs satisfy this assumption. Note that we implicitly assume that all categorical variables V i are supported on [ n i ] for some n i . It is straightforward to generalize all results to any finite support. Next, we introduce Neural Causal Models (NCMs).

Definition 3. (Neural Causal Model) A Neural Causal Model (NCM) is a special kind of SCM where U = { U 1 , · · · , U n U } ∪ { G V i : V i categorical } , all U i are i.i.d. multivariate uniform variables, G V i are i.i.d Gumblel variables and functions in (2) are Neural Networks (NNs).

The definition of NCMs we use is slightly different from that in [52] because we need to deal with mixed variables in our models. In (1), we usually take C to be the set of all SCMs. However, it is difficult to search over all SCMs since (1) becomes an infinite-dimensional polynomial programming problem. As an alternative, we can search over all NCMs. One quantity of common interest in causal inference is the Average Treatment Effect (ATE).

Definition 4. (Average Treatment Effect). For SCM M , the ATE at T = t with respect to T = t 0 is given by ATE M ( t ) = E u ∼ P ( U ) [ Y ( t ) -Y ( t 0 )] .

Partial identification can be formulated as estimating the solution to the optimization problems (1) [3]. The max and min values F and F define the interval [ F,F ] which is the smallest interval we can derive from the observed data without additional assumptions. In particular, if F = F , then the causal quantity is point-identified.

Notations. We use ∥· ∥ , ∥ · ∥ ∞ for the 1 -norm and ∞ -norm and [ n ] for { 1 , · · · , n } . Bold letters represent sets of random variables. We let F L ( K 1 , K 2 ) be the class of Lipschitz L -continuous functions f : K 1 → K 2 . We may omit the domain and use F L when the domain is clear from context. Let H ( K 1 , K 2 ) be the set of homeomorphisms from K 1 to K 2 , i.e., injective and continuous

maps in both directions. We define ϵ ( F 1 , F 2 ) = sup f 2 ∈F 2 inf f 1 ∈F 1 ∥ f 2 -f 1 ∥ for function classes F 1 , F 2 . We use standard asymptotic notation O ( · ) , Ω( · ) . Given a measure µ on R d 1 and a measurable function f : R d 1 → R d 2 , the push-forward measure f # µ is defined as f # µ ( B ) = µ ( f -1 ( B ) ) for all measurable sets B . We use P ( X ) to represent the distribution of random variable X . Let ∆ n = { ( p 1 , · · · , p n ) : ∑ n i =1 p i = 1 , p i ⩾ 0 } be the probability simplex. We use Categorical ( p ) , p ∈ ∆ n to represent categorical distribution with event probability p . We let W ( · , · ) be the Wasserstein-1 distance and S λ ( µ, ν ) be the Sinkhorn distance [12] with regularization parameter λ > 0 .

3 Approximation Error of Neural Causal Models

In this section, we study the expressive power of NCMs, which serves as a key ingredient in proving the consistency result. In particular, given an SCM M ∗ , we want to construct an NCM ˆ M such that the two causal models produce similar interventional results. Unlike in the discrete case [4, 51], latent distributions can be extremely complicated in general cases. The main challenge is how to design the structure of NCMs to ensure strong approximation power.

In the following, we first derive an upper bound on the Wasserstein distance between two causal models sharing the same causal graph. Using this result, we decompose the approximation error into two parts: the error caused by approximating structural functions via neural networks and the error of approximating the latent distributions. Then, we design different architectures for these two parts.

Decomposing the Approximation Error First, we present a canonical representation of an SCM, which essentially states that we only need to consider the case where each latent variable U i corresponds to a C 2 component of G .

Definition 5 (Canonical representation) . A SCM M with causal graph G has canonical form if

The set of latent variables consists of two sets,

where U C and g V i j are independent and g V i j are standard Gumbel variables.

The structure equations have the form

where U V i = { U C : V i ∈ C, C is a C 2 -component of G} and ( x ) k is the k -th coordinate of the vector x . We further assume that f i are normalized for categorical variables.

Given a function class F , the SCM class M ( G , F , U ) consists of all canonical SCM models with causal graph G such that f i ∈ F , i ∈ [ n V ] .

Proposition 4 in the appendix shows that any SCM satisfying Assumption 1,2 can be represented in this way and we provide an example in Appendix B.1 to illustrate how to obtain the canonical representation for a given SCM. Therefore, we restrict our attention to the class M ( G , F , U ) . For two SCM classes M ( G , F , U ) , M ( G , ˆ F , ˆ U ) , we want to study how well we can represent the models in the first class by the second class. The Wasserstein distance between the intervention distributions is used to measure the quality of the approximation. To approximate the functions in the structural equations, we need to make the following regularity assumptions on the functions.

Assumption 3. If V i is continuous, f i in (3) are L f -Lipschitz continuous. If V i is categorical, the propensity functions f i ( Pa ( V i ) , U V i ) ≜ P ( V i = j | Pa ( V i ) , U V i ) , j ∈ [ n i ] are L f -Lipschitz continuous. There exists a constant K > 0 such that max i ∈ [ n V ] ,j ∈ [ n U ] {∥ V i ∥ ∞ , ∥ U j ∥ ∞ } ⩽ K .

The following theorem summarizes our approximation error decomposition.

Theorem 1. Given any SCM model M ∈ M ( G , F L , U ) , let the treatment variable set be T = { T k } n t k =1 and suppose that Assumption 1, 2 and 3 hold for M with Lipschitz constant L , constant K .

Space-filling

Curve

...

Gumbel-Softmax

...

N

C

components of

latent dist.

W

2

Depth

L

2

W

1

Depth

L

1

Randomly output

one component

P

(

U

)

Z

i

∼

Unif(0,1)

N

C

components of

latent dist.

Z

1

Z

2

Z

N

C

...

...

For any intervention T = t and ˆ M∈M ( G , ˆ F , ˆ U ) , we have

where C G ( L, K ) is a constant that only depends on L, K and the causal graph G and f i , ˆ f i are structural functions of M and ˆ M respectively.

Theorem 1 separates the approximation error into two parts, which motivates us to construct the NCM in the following way. First, we approximate the functions f i in (3) by NNs ˆ f i . Then, we approximate the distribution of latent variables by pushing forward uniform and Gumbel variables using neural networks, i.e., ˆ U C j = ˆ g j ( Z C j ) , where { C j } are C 2 components and Z C j are multi-variate uniform and Gumbel random variables. The structural equations of the resulting approximated model ˆ M are

wheres N C,j are constants to be specified later.

For the first part, we need to approximate Lipschitz continuous functions. For simplicity, we assume that the domain of the functions are uniform cubes. Similar arguments hold for any bounded cubes. We denote NN k 1 ,k 2 ( W,L ) to be the set of ReLu NNs with input dimension k 1 , output dimension k 2 , width W and depth L . It has been shown that ϵ ( NN k 1 , 1 (2 d 1 + 10 , L 0 ) , F L ([0 , 1] d 1 , R )) ⩽ O ( L -2 /k 1 0 ) [53], where ϵ ( · , · ) denotes the approximation error defined in Section 2. For a vector valued function, we can use a wider NN to approximate each coordinate and get a similar rate.

For the second part, we approximate each U i individually by pushing forward i.i.d. multivariate uniform and Gumbel variables ˆ U C i = ˆ g i ( Z C i ) since the latent variables are independent by Assumption 1. To do so, we examine under what assumptions on the measure P over R n we can find a NN ˆ g such that W (ˆ g # λ, P ) is small, where λ ( · ) is some reference measure.

3.1 Approximating Mixed Distributions by Wide Neural Networks

In this subsection, we will extend the results in [43] to construct a wide neural network as the push-forward map. It turns out that to get a good approximation of the targeted distribution, the shape of the support is essential.

̸

Assumption 4 (Mixed Distribution) . The support of measure P has finite connected components C 1 , C 2 · · · C N C , i.e., supp ( P ) = ⋃ N C i =1 C i , and each component C i satisfies H ([0 , 1] d C i , C i ) ∩ F L ([0 , 1] d C i , C i ) = ∅ for some d C i ⩾ 0 . Recall that H ( K 1 , K 2 ) is the set of homeomorphisms from K 1 to K 2 defined at the end of Section 2.

Assumption 4 encompasses almost all natural distributions. For example, distributions supported on [0 , 1] d and closed balls, finitely supported distributions and mixtures of them all satisfy this assumption. Assumption 4 allows us to transform the support of the targeted distribution into unit cubes and the nice geometric properties of unit cubes facilitate our construction.

Now, we briefly explain the construction of the push-forward maps. An example is provided in Appendix B.1 to illustrate the construction. The NN architecture consists of three parts (see Figure 1a). The input dimension is the same as the number of connected components of the support N C . For each component C i , let H i ∈ H ([0 , 1] d C i , C i ) ∩ F L ([0 , 1] d C i , C i ) , where d C i is the dimension of component C i in Assumption 4. By P = ( H i ) # ( H -1 i ) # P on C i , we can approximate ( H -1 i ) # P first, which is supported on a unit cube. [43] constructs a wide NN ˆ g of width W and constant depth such that W (ˆ g # λ, ( H i ) -1 # P ) ⩽ O ( W -1 /d C i ) where λ is the uniform measure on [0 , 1] . Then, we approximate the Lipschitz map H i to C i to pull the distribution back to C i . These are the first two parts (yellow and blue blocks in Figure 1a) of the architecture.

Suppose that the output of i -th coordinate in the first two parts is ⃗ v i , the Gumbel-softmax layer in the third part (green box) combines different components of the support. In particular, we want to output ⃗ v i with probability p i = P ( C i ) . Let V = [ ⃗ v 1 , · · · , ⃗ v i ] , this can be achieved by outputting V X , where X = ( X 1 , · · · , X N C ) T is a one-hot random vector with P ( X i = 1) = p i . To use backpropagation in ˆ X τ i = exp((log p i + G i ) /τ ) ∑ NC exp((log p + G ) /τ ) , where τ > 0 is the temperature parameter (a hyperparameter) and

G i ∼ Gumbel (0 , 1) are i.i.d. standard Gumbel variables. As τ → 0 , the distribution of X τ converges almost surely to the categorical distribution [36, Proposition 1]. In particular, when τ = 0 , we denote ˆ X 0 i = I i =arg max j { log p i + G i } . The output of the last layer is V ˆ X τ . Note that the Gumbel-softmax function is differentiable with respect to parameter { log( p i ) } i =1 , ··· ,N C . Therefore, we can train the network with common gradient-based algorithms. Putting things together, we obtain the following theorem.

training, we use the Gumbel-Softmax trick [31] to (approximately) simulate such a random vector, k =1 k k ˆ

Theorem 2. Given any probability measure P on R d that satisfies Assumption 4, let λ be the Lebesgue measure on [0 , 1] N C , where N C is defined in Assumption 4. There exists a neural network ˆ g = ˆ g τ 3 ◦ ˆ g 2 ◦ ˆ g 1 with the above architectsure (Figure 1a) such that

Here, ˆ g i , i = 1 , 2 has the separable form ( ˆ g 1 i ( x 1 ) , · · · , ˆ g N C i ( x N C ) ) and ˆ g j 1 ∈ NN 1 ,d C j ( W 1 , Θ( d C j )) , ˆ g j 2 ∈ NN d C j ,d (Θ( d · d C j ) , L 2 ) , j ∈ [ N C ] . { d C j } are the dimension of cubes in Assumption 4. ˆ g τ 3 is the Gumbel-Softmax layer with temperature parameter τ > 0 .

Note that ˆ g τ 3 (the Gumbel-softmax layer) is actually a random function since the coefficient vector ˆ X τ is a random variable. In this sense, ˆ g # λ can be viewed as pushing forward uniform and Gumbel variables using a neural net.

3.2 Approximating Mixed Distributions by Deep Neural Networks

In this subsection, we will show that under one additional assumption on the distribution, deep ReLu networks have a stronger approximation power in approximating distributions, which means we can use fewer computational units to achieve the same worst-case theoretical approximation error.

Assumption 5 (Lower Bound) . Suppose that P is supported on a compact set K ⊂ R D , there exists a constant C f > 0 , f ∈ H ([0 , 1] d , K ) ∩ F L ([0 , 1] d , K ) , such that for any measurable set B ⊂ [0 , 1] d , P ( f ( B )) ⩾ C f λ ( B ) . Besides, if d > 0 , f # P vanishes on the boundary ∂ [0 , 1] d .

Assumption 5 implies that dλ/d ( f -1 # P ) exists and is lowered bounded by a constant C f . The next proposition extends the Skorohod representation theorem [48]. It shows that under Assumption 5, it

is possible to simulate any distribution on the unit cubes with Hölder continuous curves and uniform distribution on [0 , 1] .

Proposition 1. Let λ be the Lebesgue measure on [0 , 1] . Given any probability measure P that satisfies Assumption 5, there exists a continuous curve γ : [0 , 1] → supp ( P ) such that γ # λ = P . Furthermore, if d ⩾ 1 , γ is 1 /d -Hölder continuous.

Results from [53] show that we can approximate any Hölder continuous d -dimensional function with index α by a deep ReLu network with depth L and error O ( L -2 α/d ) . Leveraging this result, we can replace the first part of the architecture in the previous subsection with deep ReLu networks (See Figure 8 in the appendix). The remaining two parts are the same as the construction in Figure 1a. The following theorem gives a sharper bound on the approximation error compared with Theorem 2.

Theorem 3. Under the Assumption 4, if in addition, P constrained to each component satisfies Assumption 5, there exists a neural network ˆ g = ˆ g τ 3 ◦ ˆ g 2 ◦ ˆ g 1 with the above architecture such that

where d C i are the dimensions of the connected components in Assumption 4, ˆ g i , i = 1 , 2 has the form ( ˆ g 1 i ( x 1 ) , · · · , ˆ g N C i ( x N C ) ) and ˆ g j 1 ∈ NN 1 ,d C j (Θ( d C j ) , L 1 ) . ˆ g j 2 , ˆ g τ 3 , τ are the same as in Theorem 2.

Let N be the number of nonzero weights in a neural network, Theorem 3 shows that a deep NN can achieve O ( N -2 /d ) error while by Theorem 2 a wide network can only achieve O ( N -1 /d ) error. Now, we can put things together to construct NCM approximations. For simplicity, we omit the input and output dimensions of the neural network. As we mention in previous sections, our construction (5) consists of two parts, ˆ f i approximating the structural functions f i in (3), and ˆ g j ( Z j ) = ˆ g τ 3 ,j ◦ ˆ g 2 ,j ◦ ˆ g 1 ,j ( Z j ) approximating the latent variables U j . Let NCM G ( F 0 , F 1 , F 2 , τ ) be a collection of NCMs with structural equation (5) and

where N C,j is the number of connected components for U j and F i are function classes.

Corollary 1. Given a causal model M ∗ ∈ M ( G , F L , U ) , suppose that Assumptions 1-3 hold and the distributions of U C for all C 2 -component satisfy the assumptions of Theorem 3. Let d in max and d out max be the largest input and output dimension of f i in (3) and d U max be the largest dimension of all latent variables. There exists a neural causal model

with structure equations (5). For any intervention T = t , ˆ M satisfies

Similar approximation results also hold for wide NN approximation, as presented in Section 3.1. The proof can be easily adapted to the wide NNs architecture.

4 Consistency of Neural Causal Partial Identification

In this section, we prove the consistency of the max/min optimization approach to partial identification. In the finite sample setting, we consider the following estimator F n of the optimal values of (1).

where P M ∗ n , P ˆ M m n are the empirical distribution of P M ∗ , P ˆ M with sample size n, m n , µ T is some given measure and F i,n will be specified later. For example, the counterfactual outcome E [ Y (1)] is a special case of the objective. Our results can be easily generalized to any linear combination of objective functions of this form. We use the Sinkhorn distance because it can be computed efficiently in practice [15]. We want to study if F n gives a useful lower bound as n →∞ .

To match the observational distribution, we need to increase the width or depth of the NNs we use. As the sample size increases, the number of parameters also increases to infinity, which creates difficulty in the analysis. To obtain consistency, we need to regularize the functions while preserving their approximation power. Surprisingly, if we do not use any regularization, the following proposition implies that consistency may not hold even if the SCM is identifiable.

Proposition 2 (Informal, see Proposition 5 for a formal version) . There exists a constant c > 0 and an identifiable SCM M ∗ satisfying Assumptions 1-5 such that for any ϵ > 0 , there exists an SCM M ϵ satisfying W ( P M ∗ ( V ) , P M ϵ ( V )) ≤ ϵ and | ATE M ∗ -ATE M ϵ | > c .

Here, M ∗ is the ground-truth model and M ϵ are the models we use to approximate M ∗ . Proposition 2 implies that we need some regularization on M ϵ . Otherwise, even if the observation distributions are close, their ATEs can be far away. In particular, we may want to regularize the Lipschitz constant of the NN. Much work has been done to impose Lipschitz regularization during the training process [13, 50, 22, 42, 10]. We denote Lip ( f ) to be the Lipschitz constant of a function f and define the truncated Lipschitz NN class,

For simplicity, we omit the dimensions and use shorthand NN L f ,K ( W,L ) . The next theorem gives the consistency result of the min estimator. To state the theorem, we define F ∗ = E t ∼ µ T E M ∗ [ F ( V 1 ( t ) , · · · , V n V ( t ))] to be the true value and F L to be the optimal value of the following optimization problem over SCMs with Lipschitz constant L .

where the minimum is taken over M ( G , F K L , U ) with F K L = { f : ∥ f ∥ ⩽ K,f ∈ F L } and all latent distributions P ( U ) . Note that if L is the Lipschitz bound L f that we assume on our structural functions, then F L is the sharp lower bound.

Theorem 4. Let M ∗ be any SCM satisfying the assumptions of Corollary 1. Suppose that the Lipschitz constant of functions in M is L f , F : R n V → R in (6) is Lipschitz continuous and τ n > 0 , let K > 0 be the constant in Assumption 3, ˆ L f = √ d in max d out max L f ,

where d U max , d in max and d out max are defined in Corollary 1, take the radius to be α n = ϵ n + s n , ϵ n = O ( W -1 /d in max 0 ,n + ∑ 2 i =1 L -2 /d U max i,n + τ n log τ n ) ,

Note that our theorem shows that the lower limit of the solution is large than the point F ˆ L f , where ˆ L f is slightly larger than the original constraint L f we impose on the structural functions. Hence, this point can potentially be slightly smaller than the sharp bound F L f . This worsening is due to the fact that we need to use NNs that satisfy a slightly worse Lipschitz property, to ensure that we have sufficient approximation power. Although Theorem 4 does not guarantee { F n } converges to a point, it states that { F n } may oscillate in the interval [ F ˆ L f , F ∗ ] , which will still give a useful lower bound to ground-truth value F ∗ . In particular, if the graph is identifiable, we have F ∗ = F ˆ L f and { F n } converges to F ∗ . Also, note that in Theorem 4, we use wide NNs rather than deep NN for F 0 ,n because results in [53] show that wide NNs can approximate Lipschitz functions while controlling their Lipschitz constants (a property that is not yet established for deep NNs). Similar results can be obtained if wide neural nets are used for all components, invoking Theorem 2.

As a special case, we leverage Theorem 4 for a non-asymptotic rate for the ATE without confounding.

Proposition 3 (Hölder continuity of ATE) . Given two causal models M , ˆ M ∈ M ( G , F L , U ) satisfying Assumption 1 and Assumption 3, let their observational distributions be ν, µ . Suppose the norms of all variables are bounded by K > 0 . If (1) (Overlap) ν is absolutely continuous with respect to one probability measure P and the density p ν ( t | Pa ( T ) = x ) ⩾ δ > 0 for x almost surely and t ∈ [ t 1 , t 2 ] and (2) (No Confounding) there is no confounding in the causal graph G , we have

Corollary 2. Let F, µ T in Theorem 4 to be F ( V ) = Y, µ T ∼ Unif ([ t 1 , t 2 ]) , ϵ > 0 . Suppose that the assumptions in Proposition 3 and Theorem 4 hold, with probability at least 1 -O ( n -2 ) , we have | F n -F ∗ | ⩽ O ( √ α n ) , where F n , F ∗ , α n are defined in Theorem 4.

4.1 Experiments

In this section, we examine the performance of our algorithm in two settings. We compare our algorithm with the Autobounds algorithm [14] in a binary IV example in [14] and in a continuous IV model 1 . Since Autobounds can only deal with discrete models, we discretize the continuous model for comparison purposes. The implementation details are provided in the Appendix D. The setting of the first experiment is taken from [14, Section D.1]. This is a binary IV problem and we can calculate the optimal bound using LP [4]. We find that our bound is close to the optimal bound. The second experiment is a general IV example where the treatment is binary but the rest of the variables are continuous. The program that Autobounds solves after discretization contains ≈ 2 14 variables. Even with such a fine discretization, the bound obtained by Autobounds is not tighter than our NCM approach. The details of the structural equations and analysis can be found in Appendix D. We also provide an extra experiment on the counterexample of Proposition 5 in the appendix to show the effect of Lipschitz regularization.

Conclusion In this paper, we provide theoretical justification for using NCMs for partial identification. We show that NCMs can be used to represent SCMs with complex unknown latent distributions under mild assumptions and prove the asymptotic consistency of the max/min estimator for partial identification of causal effects in general settings with both discrete and continuous variables. Our results also provide guidelines on the practical implementation of this method and on what hyperparameters are important, as well as recommendations on values that these hyperparameters should take for the consistency of the method. These practical guidelines were validated with a small set of targeted experiments, which also showcase superior performance of the neural-causal approach as compared to a prior main contender approach from econometrics and statistics, that involves discretization and polynomial programming.

An obvious next step in the theoretical foundation of neural-causal models is providing finite sample guarantees for this method, which requires substantial further theoretical developments in the understanding of the geometry of the optimization program that defines the bounds on the causal effect of interest. We take a first step in that direction for the special case, when there are no unobserved confounders and view the general case as an exciting avenue for future work.

Acknowledgement Vasilis Syrgkanis is supported by NSF Award IIS-2337916 and a 2022 Amazon Research Award.

References

Joshua Angrist and Guido Imbens. Identification and estimation of local average treatment effects, 1995.

Martin Anthony, Peter L Bartlett, Peter L Bartlett, et al. Neural network learning: Theoretical foundations , volume 9. cambridge university press Cambridge, 1999.

Vahid Balazadeh, Vasilis Syrgkanis, and Rahul G. Krishnan. Partial identification of treatment effects with implicit generative models, October 2022.

Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In Uncertainty Proceedings 1994 , pages 46-54. Elsevier, 1994.

Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal of the American Statistical Association , 92(439):1171-1176, September 1997.

Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vcdimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research , 20(63):1-17, 2019.

Alexis Bellot. Towards bounding causal effects under markov equivalence. arXiv preprint arXiv:2311.07259 , 2023.

Blai Bonet. Instrumentality tests revisited, January 2013.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013.

Leon Bungert, René Raab, Tim Roith, Leo Schwinn, and Daniel Tenbrinck. CLIP: Cheap Lipschitz Training of Neural Networks. In Abderrahim Elmoataz, Jalal Fadili, Yvain Quéau, Julien Rabin, and Loïc Simon, editors, Scale Space and Variational Methods in Computer Vision , volume 12679, pages 307-319, Cham, 2021. Springer International Publishing.

Victor Chernozhukov, Han Hong, and Elie Tamer. Estimation and confidence regions for parameter sets in econometric models 1. Econometrica , 75(5):1243-1284, 2007.

Lenaic Chizat, Pierre Roussillon, Flavien Léger, François-Xavier Vialard, and Gabriel Peyré. Faster wasserstein distance estimation with the sinkhorn divergence. Advances in Neural Information Processing Systems , 33:2257-2269, 2020.

Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning , pages 854-863. PMLR, 2017.

Guilherme Duarte, Noam Finkelstein, Dean Knox, Jonathan Mummolo, and Ilya Shpitser. An automated approach to causal inference in discrete settings, September 2021.

Jean Feydy. Geometric data analysis, beyond convolutions. Applied Mathematics , 2020.

Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2681-2690, 2019.

Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability theory and related fields , 162(3-4):707-738, 2015.

Dennis Frauen, Tobias Hatt, Valentyn Melnychuk, and Stefan Feuerriegel. Estimating average causal effects from patient trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 7586-7594, 2023.

Dennis Frauen, Fergus Imrie, Alicia Curth, Valentyn Melnychuk, Stefan Feuerriegel, and Mihaela van der Schaar. A Neural Framework for Generalized Causal Sensitivity Analysis, April 2024.

David A Freedman. Statistical models and causal inference: a dialogue with the social sciences . Cambridge University Press, 2010.

Lee-Ad Gottlieb, Aryeh Kontorovich, and Robert Krauthgamer. Efficient regression in metric spaces via approximate lipschitz extension. IEEE Transactions on Information Theory , 63(8):4838-4849, 2017.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning , 110(2):393-416, February 2021.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning , 110:393-416, 2021.

Florian Gunsilius. A path-sampling method to partially identify causal effects in instrumental variable models, June 2020.

Chris H. Hamilton and Andrew Rau-Chaplin. Compact hilbert indices for multi-dimensional data. In First International Conference on Complex, Intelligent and Software Intensive Systems (CISIS'07) , pages 139-146. IEEE.

Sukjin Han and Shenshen Yang. A computational approach to identification of treatment effects for policy evaluation. Journal of Econometrics , 240(1):105680, 2024.

James J. Heckman and Edward J. Vytlacil. Instrumental variables, selection models, and tight bounds on the average treatment effect. In Wolfgang Franz, Michael Lechner, and Friedhelm Pfeiffer, editors, Econometric Evaluation of Labour Market Policies , volume 13, pages 1-15. Physica-Verlag HD, Heidelberg, 2001.

David Hilbert. Über die stetige abbildung einer linie auf ein flächenstück. Dritter Band: Analysis· Grundlagen der Mathematik· Physik Verschiedenes: Nebst Einer Lebensgeschichte , pages 1-2, 1935.

Yaowei Hu, Yongkai Wu, Lu Zhang, and Xintao Wu. A generative adversarial framework for bounding confounded causal effects. Proceedings of the AAAI Conference on Artificial Intelligence , 35(13):12104-12112, May 2021.

Yaowei Hu, Yongkai Wu, Lu Zhang, and Xintao Wu. A generative adversarial framework for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12104-12112, 2021.

Eric Jang, S. Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. ArXiv , abs/1611.01144, 2016.

Niki Kilbertus, Matt J. Kusner, and Ricardo Silva. A class of algorithms for general instrumental variable models, October 2020.

Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. arXiv preprint arXiv:1709.02023 , 2017.

Mohammad Lotfollahi, Anna Klimovskaia Susmelj, Carlo De Donno, Yuge Ji, Ignacio L Ibarra, F Alexander Wolf, Nafissa Yakubova, Fabian J Theis, and David Lopez-Paz. Compositional perturbation autoencoder for single-cell response modeling. BioRxiv , 2021.

Giulia Luise, Alessandro Rudi, Massimiliano Pontil, and Carlo Ciliberto. Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance, May 2018.

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables, March 2017.

Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review , 80(2):319-323, 1990.

Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3947-3956, 2021.

Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model. Advances in Neural Information Processing Systems , 36:32020-32060, December 2023.

Caleb H. Miles, Phyllis Kanki, Seema Meloni, and Eric J. Tchetgen Tchetgen. On Partial Identification of the Pure Direct Effect, September 2015.

Kirtan Padh, Jakob Zeitler, David Watson, Matt Kusner, Ricardo Silva, and Niki Kilbertus. Stochastic causal programming for bounding treatment effects. In Conference on Causal Learning and Reasoning , pages 142-176. PMLR, 2023.

Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgöwer. Training robust neural networks using Lipschitz bounds. IEEE Control Systems Letters , 6:121-126, 2021.

Dmytro Perekrestenko, Léandre Eberhard, and Helmut Bölcskei. High-Dimensional Distribution Generation Through Deep Neural Networks, August 2022.

Roland R. Ramsahai. Causal bounds and observable constraints for non-deterministic models. Journal of Machine Learning Research , 13(29):829-848, 2012.

Amy Richardson, Michael G. Hudgens, Peter B. Gilbert, and Jason P. Fine. Nonparametric Bounds and Sensitivity Analysis of Treatment Effects. Statistical Science , 29(4), November 2014.

Michael C. Sachs, Gustav Jonzon, Arvid Sjölander, and Erin E. Gabriel. A general method for deriving tight symbolic bounds on causal effects, December 2021.

E. Shchepin. On hölder maps of cubes. Mathematical Notes , 87:757-767, 2010.

Anatoly V Skorokhod. Limit theorems for stochastic processes. Theory of Probability & Its Applications , 1(3):261-290, 1956.

Jin Tian and Judea Pearl. On the testable implications of causal models with hidden variables, December 2002.

Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. Advances in Neural Information Processing Systems , 31, 2018.

Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressiveness, learnability, and inference, October 2022.

Kevin Xia, Yushu Pan, and Elias Bareinboim. Neural causal models for counterfactual identification and estimation. arXiv preprint arXiv:2210.00035 , 2022.

D. Yarotsky. Optimal approximation of continuous functions by very deep relu networks. ArXiv , abs/1802.03620, 2018.

Junzhe Zhang and Elias Bareinboim. Bounding causal effects on continuous outcome. Proceedings of the AAAI Conference on Artificial Intelligence , 35(13):12207-12215, May 2021.

Junzhe Zhang and Elias Bareinboim. Non-parametric methods for partial identification of causal effects. Columbia CausalAI Laboratory Technical Report , 2021.

Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data, October 2021.

A Illustration of Notions in Section 2

V

2

V

1

V

3

V

4

V

5

U

1

U

2

U

3

U

4

V

2

V

1

V

3

V

4

V

5

To further explain the notions in Section 2, we consider the following example. Let M be an SCM with the following structure equations.

where U 1 and U 2 are correlated and U 3 , U 4 and ( U 1 , U 2 ) are independent. The causal model is shown in Figure 2 and its causal graph is shown in Figure 3. In this example, U V 1 = { U 1 } , U V 2 = { U 1 , U 2 } , U V 3 = { U 2 , U 4 } , U V 4 = { U 3 , U 4 } , U V 5 = { U 3 } . Since U 1 , U 2 are correlated, C 1 = { V 1 , V 2 , V 3 } is one C 2 component because all nodes in C 1 are connected by bi-directed edges. Note that { V 1 , V 2 , V 3 , V 4 } is not a C 2 component because V 4 and V 2 is not connected by any bi-directive edge. The rest of C 2 components are C 2 = { V 4 , V 5 } , C 3 = { V 3 , V 4 } .

Now, we consider the intervention V 1 = t . Under this intervention, the structure equations can be obtained by setting V 1 = t while keeping all other equations unchanged, i.e.,

The figure of this model under intervention is shown in Figure 4.

V

2

(

t

)

V

1

(

t

)

V

3

(

t

)

V

4

(

t

)

V

5

(

t

)

U

1

U

2

U

3

U

4

B Proof of approximation Theorems

B.1 Illustration of how to construct canonical representations and neural architectures

In this section, we illustrate how to construct canonical representations and neural architectures given a causal graph via a simple example. We consider the example in the previous section. The causal graph is shown in Figure 3.

V

2

V

1

V

3

V

4

V

5

E

1

E

2

E

3

V

2

V

1

V

3

V

4

V

5

ˆ

E

1

ˆ

E

2

ˆ

E

3

Z

2

Z

1

Z

3

Following the construction in Proposition 4, we use one latent variable for each C 2 component. As we explain in Appendix A, this causal model has three C 2 components, { V 1 , V 2 , V 3 } , { V 3 , V 4 } , { V 4 , V 5 } . In the canonical representation, exactly the latent variables enter their corresponding C 2 component. The structure equation of this SCM is as follows.

If we set E 1 = ( U 1 , U 2 ) , E 2 = U 4 , E 3 = U 3 , (10) is equivalent to (8). Therefore, we can see from this example that the canonical representation does not lose any information about the SCM.

Now, we show how to construct the NCM architecture from a canonical representation. As we mentioned in Section 3, we approximate the latent distribution by pushing forward uniform and Gumbel variables. The structure equation of the NCM is

where f θ i i , g θ j j are neural networks, Z i are join distribution of independent uniform and Gumbel variables, and g θ j j has the special architecture described in Section 3.1 and Section 3.2. Figure 6 shows the architecture of the NCM. Each in-edge represents an input of a neural net.

B.2 Proof of Theorem 1

The causal graph G does not specify how latent variables influence the observed variables. There could be many ways of recovering the latent variables from a graph G . Figure 7 shows an example where two different causal models have the same causal graph. The next proposition gives a canonical representation of a causal model.

V

1

V

2

V

3

U

1

V

1

V

2

V

3

U

2

U

1

U

3

Proposition 4. Suppose that Assumption 1 and Assumption 2 hold, given any SCM M with causal graph G and latent variables U , we can construct a canonical SCM ˆ M of the form (3) by merging the latent variables in U such that M and ˆ M produce the same intervention results. Besides, functions in ˆ M have the same smoothness as M .

Proof of Proposition 4. Let G = { G V i : V i is categorical } and the latent variables in the original model M be U = { U 1 , · · · , U n U } ∪ G . By Assumption 1 and Assumption 2, U are independent and the structure equations of M have the form

where g V i k are i.i.d. standard Gumbel variables and U V i ⊂ { U 1 , · · · , U n U } contains the latent variables that affect V i . We regroup and merge the variables U i to make the model have a canonical form while not changing the functions in the structure equations. Let D 1 , · · · , D n C ⊂ V be the C 2 -component of M . For each U i , we define the vertices that are affected by U i as

We partition { U 1 , · · · , U n U } into n C sets ˆ U 1 , · · · , ˆ U n C in the following way. For each U k , U k is in the set ˆ U i if I ( U k ) ⊂ D i . If there are two components D i , D j satisfy the condition, we put U k into either of the sets ˆ U i , ˆ U j . Let ˆ U i have the same distribution as the joint distribution of the random variables in set ˆ U i . Let ˆ M∈M ( G , F , ˆ U ) the SCM with structure equations

̸

Here, we slightly abuse the notation f i . f i ignores inputs from ˆ U k 1 , · · · , ˆ U k n i that are not in U V i . Note that ˆ U 1 , · · · , ˆ U n C , G has the same distribution as U because we only merge some latent variables. In addition, the functions in the new model ˆ M has the same smoothness as the original model.

̸

We first verify that the causal graph of ˆ M is G . If there is a bi-directed arrow between nodes V i , V j in G , by the independence assumption, there must be one latent variable U k ∈ U V i ∩ U V j . There exist one D l such that I ( U k ) ⊂ D l and ˆ U l ∩ U V i = ∅ , ˆ U l ∩ U V j = ∅ . Therefore, ˆ U l will affect both V i and V j and there is a bi-directed arrow between node V i , V j in G ˆ M . Suppose that there is a bi-directed arrow between node V i , V j in G ˆ M , it means there exist a ˆ U k such that ˆ U k ∩ U V i = ∅ , ˆ U k ∩ U V j = ∅ . Let U l 1 ∈ ˆ U k ∩ U V i , U l 2 ∈ ˆ U k ∩ U V j , then

̸

Since D k is a C 2 -component, there exist a bi-directed arrow between node V i , V j in G . Therefore, G = G ˆ M .

̸

̸

Finally, we verify that M and ˆ M produce the same intervention results. The intervention distribution can be viewed as the push-forward of the latent distribution, i.e., V = F ( U ) , and the intervention operation only changes the function F . In our construction, we only merge the latent variables and leave the functions in structure equations being the same (except that they may ignore some coordinates in input). For any intervention T = t , suppose in M we have V ( t ) = F t ( U ) . Then, in ˆ M , we get ˆ V ( t ) = F t ( ˆ U ) . Since U and ˆ U has the same distribution, the distribution of V ( t ) , ˆ V ( t ) are the same.

Proof of Theorem 1. The structure equations of M have the form

where g V i k are i.i.d. standard Gumbel variables and ˆ M∈M ( G , ˆ F , ˆ U ) has structure equations

where ˆ g V i k are i.i.d. standard Gumbel variables. Let the treatment variables set be T = { T 1 , · · · , T n T } . We may give all vertices and U 0 = { U 1 , · · · , U n U } , an topology ordering (rearrange the subscript if necessary)

such that for each directed edge ( V i ( t ) , V j ( t )) , vertex V i ( t ) lies before vertex V j ( t ) in the ordering, ensuring that all edges start from vertices that appear early in the order and end at vertices that appear later in the order. We put U 0 and T at the beginning because they are the root nodes of the intervened model. We denote µ T U 0 , T , 1: k (resp. ˆ µ T U 0 , T , 1: k ) to be the distribution of U 0 , T , V 1 ( t ) , · · · , V k ( t ) (resp. ˆ U 0 , T , ˆ V 1 ( t ) , · · · , ˆ V k ( t ) ), µ T V k | U 0 , T , 1: k -1 the distribution of V k given U 0 , T , V 1 ( t ) , · · · , V k -1 ( t ) .

Let S = ∑ n V i =1 ∥ f i -ˆ f i ∥ ∞ . Next, we prove that

By definition of Wasserstein-1 distance,

For (1) , if V k is a continuous variable, we have

where we have used the Lipschitz property of g in the first inequality. If V k is categorical, let ˆ p ( pa ( v k ) , u V k ) = (ˆ p 1 , · · · , ˆ p n i ) = ˆ f ( pa ( v k ) , u V k ) and p ( pa ( v k ) , u V k ) = ( p 1 , · · · , p n i ) = f ( pa ( v k ) , u V k ) , we get

where we use ∥ V k ∥ ∞ ⩽ K,n i ⩽ K . For (2), if V i is continuous,

Since g, f k are Lipschitz continuous functions, g ( u , t , v 1 , · · · , v k -1 , f k ( pa ( v k ) , u V k )) is ( L +1) -Lipschitz continuous with respect to ( u , t , v 1 , · · · , v k -1 ) . We have

If V i is categorical,

Since g, f k are L -Lipschitz continuous functions, g ( u , t , v 1 , · · · , v k -1 , i ) is L -Lipschitz continuous with respect to ( u , t , v 1 , · · · , v k -1 ) . We have

Combine (12)-(16), we prove Equation (11). By induction, one can easily get

Let C G ( L, K ) = 2 K 2 · max { ( L +1) n V -n T -1 L , ( L +1) n V -n T } and notice that W ( µ T U 0 , T , ˆ µ T ˆ U 0 , T ) = W ( P M ( U ) , P ˆ M ( ˆ U ) ) because interventions T = t are the same, we get

B.3 Proof of results in Section 3.1

Proof of Theorem 2. Recall that the neural network consists of three parts ˆ g = ˆ g τ 3 ◦ ˆ g 2 ◦ ˆ g 1 and ˆ g 2 , ˆ g 1 have a separable form, dealing with each coordinate of the input individually. As mentioned before, each coordinate approximates the distribution of one connected component of the support. We first construct ˆ g 1 and ˆ g 2 , then use Gumbel-Softmax layer to combine each component together.

To construct the first two parts, we only need to consider each coordinate individually. For the i -th component C i , let µ i = P C i / P ( C i ) , where P C i is measure P restricted to component C i . Under Assumption 4, there exists a Lipschitz map g i 2 ∈ H ([0 , 1] d C i , C i ) ∩F L ([0 , 1] d C i , C i ) . By [43, Theorem VIII.1], there exists quantized ReLu network ˆ g i 1 of width W 1 and depth Θ( d C i ) (let s = 1 ⌈ n ( n -1) ⌉ in [43, Theorem VIII.1]), such that

where U i are i.i.d. uniform random variables on [0 , 1] . By [53, Theorem 2], there exist a deep ReLu network ˆ g i,j 2 of width Θ ( d C i ) and depth L 2 such that

Let ˆ g i 2 ( x ) = ( ˆ g i, 1 2 ( x ) , · · · , ˆ g i,d 2 ( x ) ) , we get

Thus, the width of ˆ g i 2 is Θ ( d · d C i ) . Let

where N C is the number of connected components. By Lemma 3, we get

Next, let p i = P ( C i ) and the distribution of ˆ g k 2 ◦ ˆ g k 1 ( U k ) be ˆ µ k , then we have P = ∑ N C k =1 p k µ k . By Lemma 4, we have

Finally, we analyze the error caused by the Gumbel-Softmax layer. Note that as temperature parameter τ → 0 ,

where One-hot ( k ) is N C -dimensional vector with the k -th coordinate equals to 1 and the remaining coordinates are 0 and G i ∼ Gumbel (0 , 1) are i.i.d. Gumbel distribution. Let ν τ be the distribution of w τ . By Lemma 6, for 0 < τ < 1 , we have

Let

We denote g 0 3 = lim τ → 0 g τ 3 . By Assumption 4, the support is bounded. Thus, g i 2 and ˆ g i 2 are bounded, i.e., ∥ ˆ g i 2 ( x ) ∥ ∞ ⩽ K . Since functions h ( w,X ) = w T X is Lipschitz continuous in the region ∥ w ∥ 1 = 1 , ∥ X ∥ ∞ ⩽ K , we get

from (18). Putting things together, we get

where we use (19).

B.4 Proof of results in Section 3.2

Gumbel-Softmax

Space-filling Curve

...

...

Before we prove Proposition 1, we need to introduce some important notions. Let {C n } be a sequence of partitions of the unit cube [0 , 1] d . We say {C n } has property (*) if the following properties are satisfied.

The following lemma is important in the construction of Hilbert curve.

Lemma 1 (Algorithm 3 in [25]) . Given a sequence of partitions of the unit cube {C n } that has property (*), there exist a ordering for each partition C n , i.e., O n : { 1 , · · · , 2 nd } → { i 1 i 2 · · · i n : i j = 0 , · · · , 2 d -1 } , such that the following two conclusions hold.

If ( j -1) · 2 d < i ⩽ j · 2 d , the first n digits of O n +1 ( i ) are the same as O n ( j ) .

̸

Adjacent cubes in the ordering intersect, i.e., C O n ( i ) ∩ C O n ( i +1) = ∅ . Furthermore, C O n ( i ) ∩ C O n ( i +1) is a d -1 dimensional cube.

Proof of Proposition 1. Without loss of generality, we can assume that P is supported on [0 , 1] d , vanishes at the boundary and P ( B ) ⩾ C 1 λ ( B ) for some constant C 1 > 0 and all measurable sets B ∈ [0 , 1] d . Suppose we have proven the conclusion for this case. By Assumption 5, there exists f ∈ H ([0 , 1] d , K ) ∩F L ([0 , 1] d , K ) such that f -1 # P satisfies these conditions. There exists continuous function γ such that f -1 # P = γ # λ , which implies P = ( f ◦ γ ) # λ . In particular, if γ is 1 /d -Holder continuous, f ◦ γ is also 1 /d -Holder continuous because f is Lipschitz continuous.

When d = 0 , the constant map satisfies the requirement. We focus on the case d ⩾ 1 in the following. The proof is to modify the construction of the Hilbert space-filling curve. By changing the velocity of the curve, the push-forward measure of the Hilbert space-filling curve can simulate a great number of distributions. We use λ to represent the Lebesgue measure [0 , 1] .

Proof Sketch: The construction of f is inspired by the famous Hilbert space-filling curve [28]. To illustrate the idea, let us first assume that P is absolutely continuous with respect to the Lebesgue measure λ and consider d = 2 . In the k -th step, we divide the unit cube into 2 2 k evenly closed cubes C n = { C 1 ,n , · · · , C 2 2 k ,n } such that C n is a refinement of C n -1 .

The construction of a standard Hilbert curve is to find a sequence of curves γ n that go through all the cubes in C n . The curve γ n has one special property. If γ n -1 ( t ) ∈ C k,n -1 , then γ m ( t ) ∈ C k,n -1 , ∀ m ⩾ n . For example, in Figure 9, the points on the curve in the lower-left cubes will stay in the lower-left cubes. Note that ( γ n ) # λ ( C k,n ) = λ ( γ -1 n ( C k,n ) ) is the time curve γ n stays in cubes C k,n . The idea is to change the speed of the curve so that P and ( γ n ) # λ agree on cubes in C n , i.e., P ( C k,n ) = λ ( γ -1 n ( C k,n ) ) . Since we assume that P is absolutely continuous, P ( ∂C k,n ) = 0 and we don't need to worry about how to divide the mass on the boundary. For example, let the green cubes in the Figure 9 to be C 0 and suppose that γ 1 starts from C 0 . We will change the speed so that γ 1 spends P ( C 0 ) time in this region. In the next step, we divide C 0 into four colored cubes C 1 , · · · , C 4 on the right. We change the speed again to let the time spent in each cube equal to P ( C i ) . Note that this construction preserves the aforementioned property, i.e., for t ∈ [0 , P ( C 0 )] , γ n ( t ) ∈ C 0 , ∀ n ⩾ 1 . As n →∞ , it can be proven that γ n converges uniformly to a curve γ . We can also prove that P and γ # λ agree on ∪ ∞ n =1 C n . Given that ∪ ∞ n =1 C n generate the standard Borel algebra on [0 , 1] 2 , we can conclude that P = γ # λ .

In the general case, P may not be absolutely continuous. As a result, the boundary may not be P -measure zero. We will need to perturb the cutting planes to ensure their boundaries are measured zero.

Preparation. For a n -dimensional cube C = ∏ n i =1 [ a i , b i ] , the diameter of C is diam ( C ) = √∑ n i =1 ( b i -a i ) 2 . We also define the function R that measures the ratio between maximum edge and minimum edge. For a collection of cubes C = { C 1 , · · · , C m } , C k = ∏ n i =1 [ a k i , b k i ] , R ( C ) is defined to be

R ( C ) measures the shape of the cubes in a collection. We need to control R ( C ) in the construction to obtain Holder continuity in Step 2 .

Step 1: Define partition. First, we construct a sequence of partitions that has the property (*) recursively. Let C 0 = { [0 , 1] d } . Suppose we have defined close cubes collection C n = { C i 1 ,i 2 ··· ,i n } i j =0 , ··· , 2 d -1 , , such that

C n is obtained by cutting the unit cube [0 , 1] d by planes that are paralleled to the coordinate planes.

Next, we construct C n +1 from C n that preserves these properties. We refine the division C n = { C i 1 ,i 2 ··· ,i n } i j =0 , ··· , 2 d -1 by union of hyperplane P i,r = { y : y i = r } to get { C i 1 ,i 2 ··· ,i n +1 } such that C i 1 ,i 2 ··· ,i n = ⋃ i n +1 C i 1 ,i 2 ··· ,i n +1 and diam ( C i 1 ,i 2 ··· ,i n -1 ) ⩽ 2 / 3 · diam ( C i 1 ,i 2 ··· ,i n ) . By property 1, C n is obtained by dividing [0 , 1] d using

we can further cut the unit cube using hyperplane { P i, ( r i j + r i j +1 ) / 2) } i =1 , ··· d,j =1 , ··· 2 n -1 . However, this construction may not satisfy property 3, P ( ∂C i 1 ,i 2 ··· ,i n +1 ) = 0 . We need to perturb each hyperplane to ensure a measure-zero boundary. Since

̸

is at most countable, we can perturb the hyperplanes a little bit, i.e., { P i, ( r i j + r i j +1 ) / 2+ ϵ i,j ) } i =1 , ··· d,j =1 , ··· 2 n -1 to ensure property 3. In this way, we can choose | ϵ i,j | to be sufficiently small such that

and

Therefore, R ( C n +1 ) ⩽ (1 + 2 -n ) R ( C n ) and it is easy to see that properties 1,2,3 are satisfied.

Step 2: Construct Hilbert Curve. By Lemma 1, there exist an sequence of ordering { O n } of {C n } that satisfies

The first n digits of O n +1 ( i ) with O n ( j ) , ( j -1) · 2 d < i ⩽ j · 2 d .

C O n +1 ( i ) ∩ C O n +1 ( i +1) are d -1 dimensional cubes.

Let I 0 = { [0 , 1] } , we define I n recursively. Suppose we have define interval collection I n = {{ I i 1 ,i 2 ··· ,i n } i j =0 , ··· , 2 d -1 } such that I i 1 ,i 2 ··· ,i n -1 = ⋃ i n I i 1 ,i 2 ··· ,i n and P ( C i 1 ,i 2 ··· ,i n ) = | I i 1 ,i 2 ··· ,i n | . Since P ( ∂C i 1 ,i 2 ··· ,i n +1 ) = 0 , we have P ( C i 1 ,i 2 ··· ,i n ) = ∑ i n +1 P ( C i 1 ,i 2 ··· ,i n +1 ) . With the ordering, we divide each I O n ( j ) , 0 ⩽ j ⩽ 2 nd -1 into 2 d closed sub-interval I O n +1 ( i ) , j 2 d ⩽ i < ( j +1)2 d such that I O n +1 ( i ) = [ ∑ i -1 k =1 p k , ∑ i k =1 p k ] , where p i = P ( C O n +1 ( i ) ) . This construction satisfies | I O n +1 ( i ) | = P ( C O n +1 ( i ) ) . Note that by condition 3 in step 1, P vanishes at the boundary of the cubes. Therefore,

Now, we can construct a piecewise linear function γ n +1 ( t ) such that

The idea is to construct a piecewise linear curve going through all cubes in C n +1 exactly once and modify its speed. Take v 0 ∈ Center ( C O n +1 (1) ) , v i ∈ Center ( C O n +1 ( i ) ∩ C O n +1 ( i +1) ) , v 2 d ( n +1) ∈ Center ( C O n +1 (2 d ( n +1) ) ) , where Center ( ∏ k i =1 [ a i , b i ]) = (( a i + b i ) / 2) i =1 , ··· ,k is the center of a cube. v i is well-defined since C O n +1 ( i ) ∩ C O n +1 ( i +1) are cubes of dimension d -1 by Lemma 1. One possible choice of γ n +1 ( t ) is

The first two curves γ 1 , γ 2 are shown in Figure 9. It is straightforward to verify that γ n +1 ( ∑ i k =1 p k ) = v i and

In fact, since v i -1 , v i are on different surfaces of the cube C O n +1 ( i ) , the line segment between v i -1 , v i lies inside the interior of C O n +1 ( i ) and γ n +1 goes through all the cubes C O n +1 ( i ) exactly once. We have γ -1 n +1 ( Int ( C O n +1 ( i ) )) ⊂ [ ∑ i -1 k =1 p k , ∑ i k =1 p k ] and ( γ n +1 ) # λ ( C O n +1 ( i ) ) = P ( C O n +1 ( i ) ) , where Int ( · ) is the interior of a set. Besides, we also have ( γ n +1 ) # λ ( ∂C O n +1 ( i ) ) ⩽ λ ( γ -1 n +1 ( { v j } j =1 ··· 2 d ( n +1) )) = 0 .

Finally, for any k 1 ⩾ k 2 ⩾ n, t ∈ [0 , 1] , by property 2 in step 1, (20) and (22), γ k 1 ( t ) , γ k 2 ( t ) are in one cube C i 1 ,i 2 ··· ,i n . and diam ( C i 1 ,i 2 ··· ,i n ) ⩽ ( 2 3 ) n √ d . Thus, | γ k 1 ( t ) -γ k 2 ( t ) | ⩽ ( 2 3 ) n √ d which implies { γ n ( t ) } converges uniformly to one continuous function γ ( t ) .

Step 3: Verify Conclusion P = γ # λ . By W (( γ k ) # λ, γ # λ ) ⩽ ∥ γ k -γ ∥ ∞ → 0 , ( γ k ) # λ converges weakly to γ # λ . By construction, P ( ∂C i 1 ,i 2 ··· ,i n ) = 0 and ( γ n ) # λ ( C i 1 ,i 2 ··· ,i n ) = P ( C i 1 ,i 2 ··· ,i n ) . Therefore, by condition 2 in step 1, for any k ⩾ n ,

where we use P ( ∂C i 1 ,i 2 ··· ,i n ) = 0 and ( γ k ) # λ ( ∂C i 1 ,i 2 ··· ,i n ) = 0 in the second and the last equation. We claim that γ # λ ( ∂C i 1 ,i 2 ··· ,i n ) = 0 . For k > n , we define

̸

Let k > k 1 ⩾ n and B ϵ n = { x : d ( x, ∂C i 1 ,i 2 ··· ,i n ) < ϵ } , we have

where we use (23) and the fact that ( γ k ) # λ vanishes on ∂C i 1 ,i 2 ··· ,i k in the first equality and diam ( C i 1 ,i 2 ··· ,i k 1 ) ⩽ ( 2 3 ) k 1 √ d in the last inequality. Let k → ∞ , by Portmanteau Theorem, we get

γ # λ ( ∂C i 1 ,i 2 ··· ,i n ) ⩽ γ # λ ( Int ( B k 1 )) ⩽ lim inf k →∞ ( γ k ) # λ ( B k 1 ) ⩽ lim inf k →∞ P ( B (2 / 3) k 1 √ d n ) = P ( B (2 / 3) k 1 √ d n ) . Since k 1 is arbitrary, let k 1 →∞ ,

and we conclude that γ # λ ( ∂C i 1 ,i 2 ··· ,i n ) = 0 . By Portmanteau Theorem, let k → ∞ in (23), we obtain γ # λ ( C i 1 ,i 2 ··· ,i n ) = P ( C i 1 ,i 2 ··· ,i n ) . Let C ∗ = ⋃ ∞ i =1 C i . Notice that for any open set U ∈ [0 , 1] d , U = ⋃ C ⊂ U,C ∈C ∗ C , which means the σ -algebra generated by C ∗ contains all Borel sets. Besides, C ∗ is a π -system. Hence, λ ( γ -1 ( U ) ) = P ( U ) for all Lebesgue measurable set U .

Step 4: Holder Continuity of γ . We verify the condition of [47, Theorem 1]. I n and C n in the above construction correspond to developments α k , β k in [47, Theorem 1]. Define f k ( I O k ( i ) ) = C O k ( i ) , i = 1 , · · · , 2 kd as a map from I n to C n .

By construction, if I 1 ∈ I k , I 2 ∈ I k -1 , I 1 ⊂ I 2 , f k ( I 1 ) ⊂ f k -1 ( I 2 ) .

̸

If I 1 , I 2 ∈ I k and I 1 ∩ I 2 = ∅ , I 1 , I 2 are adjacent. By Lemma 1, f k ( I 1 ) , f k ( I 2 ) share a ( d -1) -dimensional boundary. Therefore, f k ( I 1 ) ∩ f k ( I 2 ) = ∅ .

We have |C k -1 | ⩽ 2 d |C k | .

By property 3,

Let A = ∏ ∞ k =1 ( 1 + 2 -k ) . We have for any C ∈ C n , by Assumption 5

where C 1 is the constant in Assumption 5.

Since I k consists of one-dimensional intervals, we have diam ( I k ) = gap ( I k ) .

Therefore, γ is 1 /d -Holder continuous.

Proof of Theorem 3. The proof is similar to the proof of Theorem 2. The difference is that in the first part, we use a deep ReLU network to approximate the Holder continuous curve γ instead of wide ReLU network.

Let µ i = P C i / P ( C i ) . By Assumption 4 and Assumption 5, for each connected component C i of the support, there exists Lipschitz maps g i 2 ∈ H ([0 , 1] d C i , C i ) ∩ F L ([0 , 1] d C i , C i ) such that

for any measurable set B ⊂ [0 , 1] d C i . By Proposition 1, there exist a 1 /d C i -Holder continuous curve γ i : [0 , 1] → [0 , 1] d C i such that ( γ i ) # λ = ( g i 2 ) -1 # µ i .

By [53, Theorem 2] (take ω ( x ) = C ∥ x ∥ 1 /d where C is the Holder continuity constant factor), there exists deep ReLu network ˆ g i,j 1 of width 12 and depth L 1 such that

where ( γ i ) j is the j -coordinate of γ i . Let ˆ g i 1 ( x ) = ( ˆ g i, 1 1 ( x ) , · · · , ˆ g i,d 1 ( x ) ) , we get

Thus, the width of ˆ g i 1 is Θ ( d C i ) . By Lemma 3

The rest are the same as proof of Theorem 2.

̸

Proof of Corollary 1. Suppose that structure equations of M are

Let d in i be the input dimension of f i , d out i be the output dimension,

We truncate the neural network at 0 for categorical variables because the propensity functions are required to be non-negative. This truncate operation will not influence the approximation error since f i are non-negative. According to Assumption 3, f i are Lipschitz continuous. By [53, Theorem 2], we have, if V i is continuous, ∥ ˆ f i -f i ∥ ∞ ⩽ O ( L -1 /d in i 0 ) ⩽ O ( L -1 /d in max 0 ) .

ˆ

where we use Assumption 3 that n i ⩽ K and S ⩽ 1 / (2 K ) . Thus,

By Theorem 3, there exists a neural network ˆ g j with architecture in Theorem 3 such that

where Z j ∼ U ([0 , 1] N C,j ) i.i.d., N C,j is the number of connected components of support of U j , d U j is the dimension of latent variables U j and L 1 , L 2 , τ are hyperparameters of the neural network defined in Theorem 3. Let ˆ U j = ˆ g j ( Z j ) , By Lemma 5,

Let ˆ M be the casual model with the following structure equations

By Theorem 1, we get

for any intervention T = t .

C Proof of Consistency

C.1 Inconsistent Counterexample (Proposition 2)

Proposition 5. There exists a constant c > 0 and an SCM M ∗ satisfying Assumptions 1-5 with a backdoor causal graph such that there is no unobserved confounding in M ∗ . For any ϵ > 0 , there exists an SCM M ϵ with the same causal graph such that W ( P M ∗ ( V ) , P M ϵ ( V )) ≤ ϵ and | ATE M ∗ -ATE M ϵ | > c .

Proof. Let V = { X,T,Y } be the covariate, treatment and outcome respectively. T is a binary variable. Let structure equations of causal model M δ be

where all latent variables are independent and U y is mean-zero noise. The distribution of M δ is

As δ → 0 , distribution of M δ d - → M ∗ , where structure equations of M ∗ are

where U 1 , U 2 , U y are independent. The distribution of M ∗ is

It is easy to see that M ∗ satisfies Assumption 1-5. Some calculation gives W ( P M ∗ ( V ) , P M δ ( V )) ⩽ δ/ 2 . It is easy to calculate the ATE of the two models.

This example implies that even as the Wasserstein distance between P M ∗ ( V ) , P M δ ( V ) converges to zero, their ATEs do not change. As mentioned in the main body, this problem is caused by the violation of Lipschitz continuity assumption for M δ . Note that in M δ , E [ Y | X,T ] = X/δ . As δ → 0 , the Lipschitz constant explodes.

This problem may also arise in (6). If the distribution ball B n = { ˆ M : W ( P ˆ M , P M ∗ n ) ⩽ α n } includes a small ball around the true distribution S ϵ = { ˆ M : W ( P ˆ M , P M ∗ ) ⩽ ϵ } ⊂ B n and the NCM is expressive enough to approximate all the SCMs in the S ϵ , this example tells us the confidence interval may not shrink to one point as sample size increases to infinity even if the model is identifiable.

C.2 Proof of Theorem 4

To prove Theorem 4, we will need the following proposition.

Proposition 6. Given a metric space ( M,d ) and sets Θ 1 ⊂ Θ 2 · · · ⊂ Θ n ⊂ · · · ⊂ Θ ∞ ⊂ M , where Θ ∞ = ∪ ∞ n =1 Θ n , positive sequences { ϵ n } n ∈ N , { δ n } n ∈ N such that lim n →∞ ϵ n = lim n →∞ τ n = lim n →∞ δ n = 0 and continuous functions f, g n : Θ ∞ → R , suppose that

Θ ∞ is compact.

g

There exists a compact subset ˜ Θ ∞ ⊂ Θ ∞ such that for any θ 0 ∈ ˜ Θ ∞ , there exists θ ∈ Θ n such that d ( θ, θ 0 ) ⩽ ϵ n .

Consider the following optimization problems:

We assume that the feasible region of (26) and (27) are nonempty. Let f ∗ n , f ∗ , ˜ f ∗ be the minimal of (25), (26) and (27) respectively. Then, [lim inf k →∞ f ∗ n , lim sup k →∞ f ∗ n ] ⊂ [ f ∗ , ˜ f ∗ ] .

Proof of Proposition 6. By compactness of Θ ∞ and ˜ Θ ∞ , the minimizers of (26) and (27) are achievable insider these two sets. Let θ ∗ ∈ Θ ∞ , ˜ θ ∗ ∈ ˜ Θ ∞ be the minimizer of (26) and (27). We first prove that lim sup n →∞ f ∗ n ⩽ ˜ f ∗ . Note that by (24), the limiting function g is L g -Lipschitz. By condition 3, there exist θ n ∈ Θ n such that d ( θ n , ˜ θ ∗ ) ⩽ ϵ n . Note that

Therefore, θ n is a feasible point of (25). We have lim sup n →∞ f ∗ n ⩽ lim sup n →∞ f ( θ n ) = f ( ˜ θ ∗ ) = ˜ f ∗ .

Next, we argue that lim inf n →∞ f ∗ n ⩾ f ∗ . If this equation does not hold, there exists ϵ > 0 and subsequence { f ∗ n k } k ∈ N such that f ∗ n k < f ∗ -ϵ, ∀ k ∈ N . By compactness of Θ ∞ , for each k , there exists a subsequence of θ ∗ n k ∈ Θ ∞ such that θ ∗ n k satisfies constraint of (25) and f ∗ n k = f ( θ ∗ n k ) . By compactness, { θ ∗ n k } k ∈ N has a converging subseqence. Without loss of generality, we may assume that { θ ∗ n k } k ∈ N converges to ˆ θ ∗ ∈ Θ ∞ . Since g k converge uniformly to g ,

ˆ θ ∗ is a feasible point of Equation (26). Since f is continuous on Θ ,

and which leads to contradiction.

Proof of Theorem 4. We begin by defining a proper metric space. By Assumption 3, all random variables are bounded. Suppose that max i,j {∥ V i ∥ ∞ , ∥ U j ∥ ∞ } ⩽ K . A canonical causal model M∈M ( G , F , U ) is decided by

where f i are functions in the structure equations (3) and U j are uniform parts of the latent variables. We denote M θ to be the SCM represented by θ , the underlying SCM be M θ ∗ and P θ to be the distribution of M θ . We consider the space

where

d V i, in , d V i, out are the input and output dimensions of f i ,

and P ( K ) is the probability space on K . For θ = ( f 1 , · · · , f n V , P ( U 1 ) , · · · , P ( U n U )) , θ ′ = ( f ′ 1 , · · · , f ′ n V , P ( U ′ 1 ) , · · · , P ( U ′ n U ) ) , we define a metric on M

Theorem 1 states that the Wasserstein distance between two causal models is Lipschitz with respect to metric d . Now, we define Θ n . Let

In other words, P n contains all the push-forward measures of the uniform distribution by neural networks. We denote Θ n = ˆ F V 1 ,n ×··· × ˆ F V n V ,n ××P n , where

and F 0 ,n is defined in Theorem 4. Note that by construction, latent variables are independent and P n can be decomposed into direct produce P n, 1 ×··· × P n,n U . Let

and ˜ Θ ∞ = { θ ∗ } . Note that f ( θ ) is a continuous function since by Theorem 1 and the fact that F is Lispchitz continuous,

Now, we verify the conditions in Proposition 6.

By Arzelà-Ascoli theorem, F K L ( [ -K,K ] d V i, in , [ -K,K ] d V i, out ) are precompact set with respect to the infinity norm in space of continuous functions. And thus F V i are compact sets.

Since measures in P ( [ -K,K ] d U j ) are tight , P ( [ -K,K ] d U j ) are compact with respect to weak topology by Prokhorov's theorem. And the Wasserstein distance metricizes the weak topology. Thus, P ( [ -K,K ] d U j ) are compact and the space ( M,d ) is compact space. Closed set Θ ∞ ⊂ M is also compact.

2. By definition of g n ,

where we use Lemma 7 in the second inequality and triangle inequality and Theorem 1 in the third inequality. By the condition in Theorem 4, the second term (1 + log( nm n )) λ n → 0 as n →∞ . Therefore, (24) is verified with τ n = 4(1 + log( nm n )) λ n .

We then verify the uniform convergence of g n . By triangle inequality,

By Lemma 7, | S λ n ( P θ ∗ n ( V ) , P θ m n ( V ) ) -W ( P θ ∗ n ( V ) , P θ m n ( V ) ) | ⩽ 2(log( mn ) + 1) λ n and we get

where we use Theorem 1 in the last inequality.

We first bound sup θ W ( P θ ( U ) , P θ m n ( U ) ) using standard VC dimension argument. Note that { U 1 , · · · , U n U } are independent. By Lemma 5,

By the construction in Theorem 3, P θ ( U i ) ∼ ∑ j ∈ [ N C,i ] p j P ( f θ i,j ( Z i,j )) , Z i,j ∼ Unif (0 , 1) , where f θ i,j are neural networks with constant width and depth L 1 ,n + L 2 ,n , N C,i is the number of connected components of supp ( P ( U i )) and θ i,j are the parameters of the neural networks. By Lemma 4,

By [6], the pseudo-dimension of NN (Θ(1) , L ) is O ( L 2 log L ) . By boundness of all the neural networks and Lemma 8 2 , with probability at least

the following event happens.

Let

with constant C > 0 sufficiently large, we get

As long as m n = Ω( n ) , the Borel-Cantelli lemma implies that almost surely, there exists N > 0 such that when n > N , sup θ i,j W ( P ( f θ i,j ( Z i,j )) , P m n ( f θ i,j ( Z i,j ))) ⩽ O ( m -1 / ( d U max +2 ) n log m n ) for all i, j . Therefore, almost surely, for sufficiently large n ,

[17, Theorem 2] shows that for sufficiently large n , if δ n = C 1 n -1 / max { d U max , 2 } log 2 ( n ) , where C 1 > 0 is a constant,

where C > 0 is a constant. Let C 1 sufficiently large such that

Therefore, ∑ ∞ n =1 P ( W ( P θ ∗ ( U i ) , P θ ∗ n ( U i )) > δ n ) < ∞ . By Borel-Cantelli lemma, with probability 1, there exist N > 0 such that W ( P θ ∗ ( U i ) , P θ ∗ n ( U i )) ⩽ δ n , ∀ n > N . By Equation (29), with probability 1, there exist N > 0 such that

Therefore, by Equations (28), (30) and (31), almost surely, the following inequality holds eventually,

where s n is defined in Theorem 4.

[53, Proposition 1] shows that given a L -Lipschitz continuous function f : [0 , 1] m → R , there exist ˆ f ∈ NN m, 1 ( W, Θ(log( m )) such that ∥ f -ˆ f ∥ ∞ ⩽ O ( W -1 /d ) and ˆ f is √ mL -Lipschitz continuous. 3 For a multivariate function with output dimension m ′ , we can approximate each coordinate individually and get a neural network approximation ˆ f that is √ mm ′ L -Lipschitz continuous and ∥ f -ˆ f ∥ ∞ ⩽ O ( W -1 /m ) .

Next, we define the truncate operator T K f = min { K, max {-K,f }} , which is a contraction mapping, and prove that applying the truncate operator will not increase approximation error. If | f | ⩽ K , we have

̸

This function can be realized by feed-forward network with width O ( m 2 ) and depth O (log m ) .

Therefore, we get that

Theorem 3 implies that for each latent variable U i , there exist a neural network g i with architecture in Corollary 1 such that

By Assumption 3, we know ∥ U i ∥ ∞ ⩽ K , which implies

Combining Equations (32) and (33) with the same proof as Corollary 1, it can be proven in the same way as Corollary 1 that there exists a θ n ∈ Θ n satisfying

Let ˜ Θ ∞ = { θ ∗ } , we have verified the third assumption in Proposition 6.

Take the Wasserstein radius to be α n = O ( s n + ϵ n ) , Proposition 6 implies the conclusion.

C.3 Proof of Proposition 3

Lemma 2. Let ( ˆ T, ˆ Y ) ∼ µ, ( T, Y ) ∼ ν and suppose that f ( t ) = E ν [ Y | T ] , ˆ f ( t ) = E µ [ ˆ Y | ˆ T ] are L -Lipschitz continuous and | f ( t ) | ⩽ K, | ˆ f ( t ) | ⩽ K , then we have

where C W = 4 LK +2 K max { L, 1 } .

Proof of Lemma 2. By the duality formulation of Wasserstein-1 distance, we have

Let g 0 ( t, y ) = ( ˆ f ( t ) -f ( t )) y , we verify g 0 is a Lipschitz continuous function in { ( t, y ) : ∥ ( t, y ) ∥ ∞ ⩽ K } .

Let L g = 2 K max { L, 1 } , we have proven that g 0 is L g -Lipschitz continuous in { ( t, y ) : ∥ ( t, y ) ∥ ∞ ⩽ K } . Thus,

Now, let h ( t ) = ( ˆ f ( t ) -f ( t )) ˆ f ( t ) . Following the same argument, it can be proven that h ( t ) is Lipschitz continuous with Lipschitz constant being L h = 4 LK . Hence,

Plug into (34), and we get

Proof of Proposition 3. If Y is not descendant of T , E [ Y ( t )] = E [ Y ] and we have

Now, suppose that Y is a descendant of T . Let X = Pa ( T ) . Note that f y ( x, t ) = E M [ Y | X = x, T = t ] is L y -Lipschitz continuous with Lipschitz constant L y ⩽ ( L + 1) n V . This is because Y can be written as Y = F 0 ( X,T, U y ) where U y are latent variables independent of T, X and F 0 is composition of f i in structure equations. The composition of Lipschitz functions is still Lipschitz and F 0 is a composition of at most n V L -Lipschitz functions. F 0 is ( L +1) n V -Lipschitz and so is f y ( x, t ) = E U y [ F 0 ( X,T, U y )] . Recall that ν, µ are the observation distributions of M and ˆ M respectively in Proposition 3. Similarly, ˆ f y ( x, t ) = E ˆ M [ ˆ Y | X = x, T = t ] is L y -Lipschitz continuous. By Lemma 2 and the overlap assumption, we have

Note that ˆ f y ( x, t ) is Lipschitz continuous, we have

Therefore,

Combine all the equations, we get

Proof of Corollary 2. In the proof of Theorem 4, we know with probability at least 1 -O ( n -2 ) , (6) has feasible solutions and

where notations θ and P θ are defined in the proof of Theorem 4 and θ n is one of the minimizers of (6). By Lemma 7, we know that

We get that

By Proposition 3, we get | F n -F ∗ | ⩽ O ( α n ) .

D Experiments

The structure equations of the generative models are (5). We use three-layer feed-forward neural networks with width 128 for each ˆ f i and six-layer neural networks with width 128 for each ˆ g j . We use the Augmented Lagrangian Multiplier (ALM) method to solve the optimization problems as in [3]. We run 600 epochs and use a batch size of 2048 in each epoch. m n is set to be m n = n . The "geomloss" package [16] is used to calculate the Sinkhorn distance. To impose Lipschitz regularization, we use the technique from [23] to do layer-wise normalization to the weight matrices with respect to infinity norm. The upper bound of the Lipschitz constant in each layer is set to be 8 . The τ in the Gumbel-softmax layer is set to be 0 .

For the choice of Wasserstein ball radius α n , we use the subsampling technique from [11, Section 3.4]. We can take the criterion function in [11] to be Q ( θ ) = W ( P M ∗ ( V ) , P M θ ( V )) and its empirical estimation to be Q n ( θ ) = W ( P M ∗ n ( V ) , P M θ n ( V )) . In the first step, we minimize the Wasserstein distance ˆ θ ∗ n = arg min θ W ( P M ∗ n ( V ) , P M θ n ( V )) . Then, [11] propose to refine the radius Q n ( ˆ θ ∗ n ) by taking the quantile of subsample { sup θ : Q n ( ˆ θ ) ⩽ βQ n ( ˆ θ ∗ n ) Q j,b ( θ ) } , where Q j,b denotes the criterion function estimated at j -th subsample of size b . However, it is time-consuming to solve this optimization problem many times. In practice, we set the radius α n to be the 95% quantile of { W ( ¯ P M ∗ j } ( V ) , P M ˆ θ ∗ n n ( V )) , where we use 50 subsamples ¯ P M ∗ j , j = 1 , · · · , 50 from P M ∗ n ( V ) with size b = ⌊ 15 √ n ⌋ .

D.1 Discrete IV [14]

We consider the noncompliance binary IV example in [14, Section D.1]. The causal graph is shown in Figure 10. We could see from Table 2 that the bound given by NCM is slightly worse than Autobounds but is still close to the optimal bound. Besides, the Autobounds bound does not cover the optimal bound in this example.

Z

T

Y

U

D.2 Continuous IV

Now, we turn to the continuous IV setting, where T is still binary, but Y and Z can be continuous. Let E λ i ∼ λZ 1 +(1 -λ ) Unif ( -1 , 1) , where Z 1 is the Gaussian variable conditioning on [ -1 , 1] and the structure equations of M λ to be

where E λ i are independent. It is easy to see the ATE of this model is 3 regardless of λ . In the experiment, we randomly choose ten λ ∼ Unif (0 , 1) . For each λ , we run the algorithms 5 times to get the bound. We choose different latent distributions (indexed by λ ) in the experiments to create more difficulty.

Since Autobounds can only deal with discrete variables, we discretize Z and Y . Suppose that Z ∈ [ l, u ] , we map all points in intervals [ l + i ( u -l ) /k, l +( i +1)( u -l ) /k ] , I = 0 , 1 · · · , k -1 to the middle point l +( i +1 / 2)( u -l ) /k . We choose k = 8 in the following experiments. The choice will give rise to polynomial programming problems with 2 14 decision variables, which is quite large.

Table 3 demonstrate the results. While both algorithms cover the true ATE well, we can see that NCM gives much tighter bounds on average. The main reason may be that the discretized problem does not approximate the original problem well enough. It is possible that a larger discretized parameter k can give a better bound, but since the size of the polynomial problem grows exponentially with k , the optimization problem may be intractable for large k . On the contrary, NCM does not suffer from computational difficulties.

D.3 Counterexample

We test our neural causal method on the counterexample in Appendix C.1. We choose the noise U y to be the normal variable. The structure equations are

We compare the confidence intervals of the unregularized neural casual method and the Lipschitz regularized one under different architectures. We choose the layerwise Lispchitz constants upper bound to be 5 and 1 . 2 . As a benchmark, we also include the bound obtained by the Double Machine Learning estimator using the EconML package. The result is shown in Appendix D.3. For this identifiable case example, the double ML estimator produces better bounds for the ATE. The intervals given by regularized and unregularized NCM are similar to the regularized one slightly better in the left figure, where we use medium-sized NNs. However, in the right figure, the obtained intervals after regularization are tighter, although slightly biased, compared with the regularized setting. From these two experiments, we conclude that the architecture of NNs will also influence the results and adding regularization during the training process can prevent extreme confidence intervals or inconsistency.

Counterexample

2000

Sample Size

Counterexample

1.50

True ATE

Unegularized NCM

1.25

Regularized NCM Lip-1.2

+

Double ML

1.00

0.75

ATE

0.50

0.25

0.00

0.25

500

1000

2000

5000

Sample Size

E Technical Lemmas

Lemma 3. Let µ, ˆ µ be two measures on compact set K ⊂ R d 1 , for any measurable functions F, ˆ F : K → R d 2 , if F is L -Lipschitz continuous, then

Proof. For any 1 -Lipschitz function g : R d 2 → R ,

Note that for any x, y ∈ R d 1 ,

Thus, g ◦ F is L -Lipschitz continuous and

ATE

1.5

1.0

0.5

0.0

0.5

500

1000

For the second term,

Combine (35) and (36), we have the conclusion.

True ATE

Unegularized NCM

Regularized NCM Lip-5

Double ML

5000

Lemma 4. Let µ 1 , · · · , µ n , ˆ µ 1 , · · · , ˆ µ n be measure on R d , for any p 1 , · · · , p n ∈ [0 , 1] such that ∑ n k =1 p k = 1 , we have

Proof. For any 1 -Lipschitz function f : R d → R , we have

By dual formulation of Wasserstein-1 distance,

Lemma 5. Given two measures µ = µ 1 ⊗··· ⊗ µ n , ˆ µ = ˆ µ 1 ⊗··· ⊗ ˆ µ n , we have

Proof. For any ϵ > 0 , let π k be a coupling of µ k , ˆ µ k such that

Then,

Let ϵ → 0 , we get the result.

Lemma 6 (Approximation error of Gumbel-Softmax layer) . Given 1 > τ > 0 and p 1 , · · · , p n > 0 , let µ τ ∼ X τ = ( X τ 1 , · · · , X τ n ) be

where G k ∼ exp( -x -exp( -x )) are i.i.d. standard Gumbel variables. Let µ 0 be the distribution of X 0 = lim τ → 0 X τ = ( X 0 1 , · · · , X 0 n ) . Then,

Proof. Without loss of generality, we may assume that ∑ n k =1 p k = 1 . Otherwise, we can divide the denominator and numerator in (37) by ∑ n k =1 p k . Let ∆ n = { ( x 1 , · · · , x n ) : ∑ n k =1 x k = 1 } . We construct a transportation map T : ∆ n → ∆ n as follows. T ( x 1 , · · · , x n ) is a n -dimensional vector with all zeros except that the i = arg max j x j -th entry being one. If there are multiple coordinates

that is maximum, we choose the first coordinate and let the remaining coordinate be zero. It is easy to verify that T # µ τ and µ 0 have the same distribution [36]. For any δ > 0 , we have

where we have used exp( -x -exp( -x )) ⩽ exp( -1) . Let event E δ = {∃ i, j ∈ [ n ] : | G i +log p i -( G j +log p j ) | < δ } , we have

̸

Now, we calculate the transportation cost

Note that ∥ T ( X τ ) -X τ ∥ 1 ⩽ 2 . For the first term, we have

For the second term, under event E c δ , let k max = arg max i X i , if j = k max = arg max i (log p k + G k )

̸

̸

If j = k max , we have

Therefore,

We get an upper bound of the transportation cost,

Let δ = -τ log τ , we get

Lemma 7 (Approximation Error of Sinkhorn distance) . For any µ ∈ ∆ n , ν ∈ ∆ m and λ > 0 , we have

where W 1 ,λ ( · , · ) is the entropy regularized Wasserstein-1 distance. Moreover, we have the following estimation of approximation error.

̸

̸

where c = max { h ( T ) : transportation plan T is achieve optimal loss W ( µ, ν ) } . Since T ij ∈ [0 , 1] and f ( x ) = -x log x is concave, we have

By definition of Sinkhorn distance,

By (38), we get

Lemma 8. Given a measure µ on R d and a real function class F with output dimension d , suppose that the pseudo-dimension of F is less than d F < ∞ and there exists K > 0 such that | f ( x ) | ⩽ K, ∀ f ∈ F , x ∈ R d ′ , then with probability at least 1 -exp ( O ( δ -d log ( δ -1 ) + d F log ϵ -1 -nϵ 2 )) ,

for all δ, ϵ > 0 , where µ n is the empirical distribution of µ .

Proof. By the dual formulation of the Wasserstein distance,

We define

where N ( ϵ, S, ∥· ∥ ) is the covering number of set S in ℓ norm. Obviously, if h is 1-Lipschitz,

By Theorem 18.4 in [2], for any fixed h ,

By Theorem 17.1 in [2],

Let H δ be the δ -net of the set H = { h : h ∈ F 1 ( [ -K,K ] d , [ -K,K ] d ) , h (0) = 0 } in ℓ ∞ norm. By Lemma 6 in [21],

Therefore, with probability no more than exp ( O ( δ -d log δ -1 + d F log ϵ -1 -nϵ 2 ))

Notice that

which implies that with probability no more than exp ( O ( δ -d log δ -1 + d F log ϵ -1 -nϵ 2 ))

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately summarize our contributions in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mention some limitations as future directions in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions and detailed proof are given.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all implementation details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all implementation details in the appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all implementation details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the confidence interval.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide all implementation details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the Code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:This is a theoretical work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies theoretically the partial identification capabilities of neural causal models (NCMs), that is, the extent to which this type of models can approximate the interval on which a certain causal query falls on, $\\theta(\\mathcal{M}) \\in [\\underline{F}, \\overline{F}]$. To this end, the authors develop a lot of results showing that under ideal scenarios with oracle knowledge on the exogenous distributions, NCMs can approximate any SCM if the proper architecture and regularization is used. Interestingly, the authors show that regularization is key in these settings, as otherwise the observational likelihood can be modelled arbitrarily well, while the interventional distribution do not look alike. It is worth-noting that the theory developed consider general scenarios with confounders and mixed-type data.\n\nSTRENGTHS:\n- **S1.** This work addresses an intellectually interesting question of whether NCMs can properly perform causal partial identification.\n- **S2.** The theory developed looks general, sound, and it makes sense on an intuitive level (disclaimer: I have _not_ looked into the proofs).\n- **S3.** The work properly studies mixed-type data, which I have rarely seen done.\n- **S4.** The fact that NCMs _need_ a form of regularization to properly approximate the SCMs is really interesting.\n\nWEAKNESSES:\n- **W1.** My personal main concern is that I do not immediately see any practical implication of the theory developed here. What I mean is that, while I find it interesting, if I have understood it correctly it works under the assumption that we know the exogenous distribution of the ground-truth model, which we never know, and which we could be arbitrarily different from whatever we model (we can always change the exogenous distribution and push the transformation to the generator $f$).\n- **W2.** While trying to be general, I similarly feel that some of the assumptions can be quite restrictive in practice. E.g., global Lipschitz-continuity is quite a constraining assumption from my point of view.\n- **W3.** I know it is a dense theory paper, but the presentation leaves a lot to be desired. There is a lot of similar notation, non-standard definitions (e.g. SCMs), and a lack of intuition/examples that makes the results hard to parse. The paper is simply not accessible to a large percentage of the community.\n- **W4.** The experimental section is extremely short, and it would require of more settings, an ablation study of the different components of the theory, and a proper analysis of the results.\n\nQUESTIONS:\nI have no specific questions, but I want to make clear that a dense paper like this would require many more hours for me to fully understand it and I may have missed some important points. I will reflect this in my confidence score.\n\nLIMITATIONS:\nLimitations are **not** discussed. In the checklist, it is said to be discussed on the Conclusion section, which does not exist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper provides a novel perspective and solid contribution to the case where continuous and categorical variables both exist.

STRENGTHS:
The presentation and organization are concise and clear; the contribution is solid.

WEAKNESSES:
The assumption is somehow strong (e.g., assumption 2).

QUESTIONS:
How do we argue the importance of solving the problem of ``the approximation error and consistency of the optimization-based approach to partial identification via NCMs''?

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends neural causal models (NCMs) to the continuous and mixed-type variables. NCMs is a neural-networks based tool to perform an automated point/partial identification and estimation of causal queries. The authors provided new theoretic results (1) on how to construct a canonical representation of an SCM, (2) on how to approximate it with sufficiently expressive Lipshitz neural networks, and (3) on additional assumptions to perform the consistent identification/estimation of an average treatment effect (ATE) from an arbitrary causal digram and an observational distribution.  The paper provides experimental results for the partial identification of the ATE in the discrete and continuous outcomes settings.\n\nSTRENGTHS:\nTo the best of my knowledge, this paper is the first one to advance neural causal models (NCMs) to the mixed-type variables setting. NCMs are a universal tool for general automated partial identification and estimation of causal quantities (a very important problem in causal inference and treatment effect estimation). The theory provided in the paper is very general and extends over simple linear SCMs. The paper is clearly written (as far as a massive theoretic contribution allows) and well-structured. The authors rigorous proofs for all the theoretic statements.\n\nWEAKNESSES:\nI didn’t find any major or minor weaknesses in the paper. Yet, I have several suggestions on how to improve the paper’s understandability for the general public of the NeurIPS conference:\n1. I encourage the authors to provide more examples of how to construct canonical representations for some simple SCMs and then the corresponding neural architectures of the NCMs.\n2. I would provide a more precise explanation of Figures 1 and 5, e.g., the number of layers in the yellow and blue blocks. Also, multiple notation elements could potentially be added to the Figures, e.g., the number of connected components in the latent space or the number of confounded components.\n3. I also encourage the authors to provide the code of the proposed method.\n4. Seems like the conclusion is missing in the main part of the text. \n \nI am willing to increase my score if the authors implement the above-mentioned suggestions.\n\nQUESTIONS:\nI appreciate the provided continuous IV experiment. I wonder, what is the performance of the method in other confounded ATE settings where the ground-truth is available, e.g., a no-assumptions bound for the ATE with hidden confounding?\n\nLIMITATIONS:\nThe authors have fully stated the limitations of their theory.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develop consistency results for partial identification via neural causal model with both continuous and categorical variables. Their results shed light on the impact of the neural network architecture and Lipschitz regularization during training. The resulting method can be trained via gradient-based optimization algorithms, and is validated on synthetic data.\n\nSTRENGTHS:\n- The paper is clearly written. The assumptions used are clearly stated.\n- The identification result is useful and a good complement to existing ones, by making them more general.\n- The identification result and algorithm are technically sound.\n\nWEAKNESSES:\n- The empirical validation of the algorithm/result is not extensive, but I understand that the goal of the work is to establish identification results and a thorough empirical study might not be necessary.\n- The paper would benefit from giving some proof sketches (or a brief overview of the proof strategy) in the main paper.\n- Many of the assumptions and results are not well explained, making it not straightforward to understand the assumptions and implications of the results.\n\nQUESTIONS:\n- A conclusion section does not seem to be provided.\n- It would enrich the paper to briefly discuss the connection with optimization-based approaches in causal learning (many of which also used similar techniques including augmented Lagrangian and gumbel techniques), although the task is not completely the same (see e.g. https://arxiv.org/abs/2007.01754, https://arxiv.org/abs/1910.08527, https://openreview.net/forum?id=HsSLdHuAmnY).\n- It would be helpful to elaborate more on the assumptions required and discuss some examples where the assumptions are satisfied/violated.\n- Other related work that could be worth mentioning: https://arxiv.org/abs/2105.12891\n\nLIMITATIONS:\nThe limitations have not been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}