Title: Score matching through the roof: linear, nonlinear, and latent variables causal discovery
Venue: NeurIPS 2024
Accepted: no
Keywords: Causal discovery

score matching

latent variables
Abstract: Causal discovery from observational data holds great promise, but existing methods rely on strong assumptions about the underlying causal structure, often requiring full observability of all relevant variables. We tackle these challenges by leveraging the score function $\nabla \log p(X)$ of observed variables for causal discovery and propose the following contributions. First, we generalize the existing results of identifiability with the score to additive noise models with minimal requirements on the causal mechanisms. Second, we establish conditions for inferring causal relations from the score even in the presence of hidden variables; this result is two-faced: we demonstrate the score's potential as an alternative to conditional independence tests to infer the equivalence class of causal graphs with hidden variables, and we provide the necessary conditions for identifying direct causes in latent variable models. Building on these insights, we propose a flexible algorithm for causal discovery across linear, nonlinear, and latent variable models, which we empirically validate.

Abstract

Causal discovery from observational data holds great promise, but existing methods rely on strong assumptions about the underlying causal structure, often requiring full observability of all relevant variables. We tackle these challenges by leveraging the score function ∇ log p ( X ) of observed variables for causal discovery and propose the following contributions. First, we generalize the existing results of identifiability with the score to additive noise models with minimal requirements on the causal mechanisms. Second, we establish conditions for inferring causal relations from the score even in the presence of hidden variables; this result is two-faced: we demonstrate the score's potential as an alternative to conditional independence tests to infer the equivalence class of causal graphs with hidden variables, and we provide the necessary conditions for identifying direct causes in latent variable models. Building on these insights, we propose a flexible algorithm for causal discovery across linear, nonlinear, and latent variable models, which we empirically validate.

1 Introduction

The inference of causal effects from observations holds the potential for great impact arguably in any 16 domain of science, where it is crucial to be able to answer interventional and counterfactual queries 17 from observational data [1, 2, 3]. Existing causal discovery methods can be categorized based on 18 the information they can extract from the data [4], and the assumptions they rely on. Traditional 19 causal discovery methods (e.g. PC, GES [5, 6]) are general in their applicability but limited to the 20 inference of an equivalence class. Additional assumptions on the structural equations generating 21 effects from the cause are, in fact, imposed to ensure the identifiability of a causal order [7, 8, 9, 10]. 22 As a consequence, existing methods for causal discovery require specialized and often untestable 23 assumptions, preventing their application to real-world scenarios. 24

25

26

27

28

29

30

31

32

Further, the majority of existing approaches are hindered by the assumption that all relevant causes of the measured data are observed, which is necessary to interpret associations in the data as causal relationships. Despite the convenience of this hypothesis, it is often not met in practice, and the solutions relaxing this requirement face substantial limitations. The FCI algorithm [11] can only return an equivalence class from the data. Appealing to additional restrictions ensures the identifiability of some direct causal effects in the presence of latent variables: RCD [12] relies on the linear non-Gaussian additive noise model, whereas CAM-UV [13] requires nonlinear additive mechanisms. Nevertheless, the strict conditions on the structural equations hold back their applicability to more general settings.

33

34

35

36

Our paper tackles these challenges and can be put in the context of a recent line of work that derives a connection between the score function

∇

data-generating process [14, 15, 16, 17, 18, 19].

log

p

(

X

)

and the causal graph underlying the

The use of the score for causal discovery is practically appealing, as it yields advantages in terms of scalability to high dimensional graphs [16]

and guarantees of finite sample complexity bounds [20]. Instead of imposing assumptions that ensure 37 strong, though often impractical, theoretical guarantees, we organically demonstrate different levels of 38 identifiability based on the strength of the modeling hypotheses, always relying on the score function 39 to encode all the causal information in the data. Starting from results of Spantini et al. [21] and Lin 40 [22], we show how constraints on the Jacobian of the score ∇ 2 log p ( X ) can be used as an alternative 41 to conditional independence testing to identify the Markov equivalence class of causal models with 42 hidden variables. Further, we prove that the score function identifies the causal direction of additive 43 noise models, with minimal assumptions on the causal mechanisms. This extends the previous findings 44 of Montagna et al. [17], limited by the assumption of nonlinearity of the causal effects, and Ghoshal 45 and Honorio [14], limited to linear mechanisms. On these results, we build the main contributions 46 of our work, enabling the identification of direct causal effects in hidden variables models. 47

48

49

50

51

52

53

54

55

Our main contributions are as follows: (i) We present the necessary conditions for the identifiability of direct causal effects and the presence of hidden variables with the score in the case of latent variables models. (ii) We propose AdaScore (Adaptive Score-based causal discovery), a flexible algorithm for causal discovery based on score matching estimation of ∇ log p ( X ) [23]. Based on the user's belief about the plausibility of several modeling assumptions on the data, AdaScore can output a Markov equivalence class, a directed acyclic graph, or a mixed graph, accounting for the presence of unobserved variables. To the best of our knowledge, the broad class of causal models handled by our method is unmatched by other approaches in the literature.

56

57

58

59

60

61

62

63

64

65

66

67

68

69

2 Model definition and related works

In this section, we introduce the formalism of structural causal models (SCMs), separately for the the cases with and without hidden variables.

2.1 Causal model with observed variables

Let X be a set of random variables in R defined according to the set of structural equations

N i ∈ R are mutually independent random variables with strictly positive density, known as noise or error terms. The function f i is the causal mechanism mapping the set of direct causes X PA G i of X i and the noise term N i , to X i 's value. A structural causal model (SCM) is defined as the tuple ( X,N, F , P N ) , where F = ( f i ) k i =1 is the set of causal mechanisms, and P N is the joint distribution relative to the density p N over the noise terms N ∈ R k . We define the causal graph G as a directed acyclic graph (DAG) with nodes X = { X 1 , . . . , X k } , and the set of edges defined as { X j → X i : X j ∈ X PA G i } , such that PA G i are the indices of the parent nodes of X i in the graph G . (In the remainder of the paper, we adopt the following notation: given a set of random variables Y = { Y 1 , . . . , Y n } and a set of indices Z ⊂ N , then Y Z = { Y i | i ∈ Z, Y i ∈ Y } .)

Under this model, the probability density of X satisfies the Markov factorization (e.g. Peters et al. 70 [1] Proposition 6.31): 71

where we adopt the convention of lowercase letters referring to realized random variables, and use p to denote the density of different random objects, when the distinction is clear from the argument. This factorization is equivalent to the global Markov condition (e.g. Peters et al. [1] Proposition 6.22) that demands that for all { X i , X j } ∈ X,X Z ⊆ X \ { X i , X j } , then

where ( · | = · |· ) denotes probabilistic conditional independence of X i , X j given X Z , and ( · | = d G · |· ) 72 is the notation for d-separation , a criterion of conditional independence defined on the graph G 73 (Definition 5 of the appendix). As it is commonly done, we assume that the reverse direction 74 X i | = X j | X Z = ⇒ X i | = d G X j | X Z hold, and we say that the density p is faithful to the graph G 75 [2, 24] (hence the faithfulness assumption ). Together with the global Markov condition, faithfulness 76 implies an equivalence between the probabilistic and graphical notions of conditional independence: 77

In general, several DAGs may entail the same set of d-separations: graphs sharing such common 78 structure form a Markov equivalence class (see Definition 6 in the appendix). 79

The above model assumes that there aren't any unobserved causes of variables in X , other than the 80 noise terms in N . As we are interested in distributions with potential hidden variables, we will now 81 generalize our model to represent data-generating processes that may involve latent causes. 82

83

84

85

86

Definitions on graphs. As graphs play a central role in our work, Appendix A.1 provides a detailed overview of the fundamental notation and definitions that we rely on in the remainder of the paper. For the next section, we advise the reader to be comfortable with the notions of ancestors (Definition 2) and inducing paths (Definition 3) in DAGs.

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

Closely related works. Several methods for the causal discovery of fully observable models using the score have been recently proposed. Ghoshal and Honorio [14] demonstrates the identifiability of the linear non-Gaussian model from the score, and it is complemented by Rolland et al. [15], which shows the connection between score matching estimation of ∇ log p ( X ) and the inference of causal graphs underlying nonlinear additive noise models with Gaussian noise terms, also allowing for sample complexity bounds [20]. Montagna et al. [17] provides identifiability results in the nonlinear setting, without posing any restriction on the distribution of the noise terms. Montagna et al. [16] is the first to show that the Jacobian of the score provides information equivalent to conditional independence testing in the context of causal discovery, limited to the case of additive noise models. All of these studies make specialized assumptions to find theoretical guarantees of identifiability, whereas our paper provides a unifying view of causal discovery with the score function, which generalizes and expands the existing results.

2.2 Causal model with unobserved variables

Under the model (1), we consider the case where the set of variables X is partitioned into the disjoint subsets of observed random variables V = { V 1 , . . . , V d } and unobserved (or latent ) random variables U = { U 1 , . . . , U p } . We assume that the following set of structural equations is satisfied:

where U i stands for the set of unobserved parents of V i , and V PA G i = { V k | k ∈ PA G i , V k ∈ V } are the observed direct causes of V i . Some of the causal relations and the conditional independencies implied by the set of equations (4) can be summarized in a graph obtained as a marginalization of the DAG G onto the observable nodes V .

Definition 1 (Marginal graph, Zhang [25]) . Let X = V ˙ ∪ U and G be a DAG over X . The following construction gives the marginal graph M G V , with nodes V and edges found as follows:

pair of nodes V i , V j are adjacent in the graph M G V if and only if there is an inducing path between them relative to U in G ;

for each pair of adjacent nodes V i , V j in M G V , orient the edge as V i → V j if V i is an ancestor of V j in G , else orient it as V i ↔ V j .

We define the map G ↦→ M G V as the marginalization of the DAG G onto V , the observable nodes.

The graph resulting from the above construction is a maximal ancestral graph (MAG, Definition 4), hence we will often refer to it as the marginal MAG of G . Intuitively, a directed edge denotes the presence of an ancestorship relation, whereas bidirected edges represent dependencies that can not be removed by conditioning on any of the variables in the graph.

In the case of DAGs, d-separation encodes the probabilistic conditional independence relations between the variables of X in the graph G , as explicit by Equation (3). Such notion of graphical separation has a natural generalization to maximal ancestral graphs, known as m-separation (Definition 5 of the appendix). Zhang [25] shows that m-separation and d-separation are in fact equivalent (see Lemma 1 of the appendix), such that given V Z ⊂ V and { V i , V j } ⊂ V , the following holds:

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

where ( · | = m M G V · | · ) denotes m-separation relative to the graph M G V . Just like with DAGs, MAGs that imply the same set of conditional independencies define an equivalence class. Usually, the common structure of these graphs is represented by partial ancestral graphs (PAGs, Definition 7 of the appendix). We use P M G V to denote the PAG relative to M G V .

Problem definition. In this work, our goal is to provide theoretical guarantees for the identifiability of the Markov equivalence class of the marginal graph M G V and its direct causal effects with the score, where variables V i are defined according to Equation (4).

Without further assumptions on the data-generating process, we can identify the graph M G V only up to its partial ancestral graph, as discussed in the next section.

Closely related works. Causal discovery with latent variables have been first studied in the context of constraint-based approaches with the FCI algorithm [11], which shows the identifiability of the equivalence class of a marginalized graph via conditional independence testing. The RCD and CAM-UV [12, 13] approaches instead demonstrate the inferrability of directed causal edges via regression and residuals independence testing. Both methods rely on strong assumptions on the causal mechanisms: their theoretical guarantees apply to models where the effects are generated by a linear (RCD) or nonlinear (CAM-UV) additive contribution of each cause. Our work demonstrates that using the score function for causal discovery unifies and generalizes these results, presenting an alternative to conditional independence testing for constraint-based methods, and being agnostic about the class of causal mechanisms of the observed variables, under the weaker requirement of additivity of the noise terms.

3 Theory for a score-based test of separation

In this section, we show that for V ⊆ X generated according to Equation (4) the Hessian matrix of log p ( V ) identifies the equivalence class of the marginal MAG M G V . It has already been proven that cross-partial derivatives of the log-likelihood are informative about a set of conditional independence relationships between random variables: Spantini et al. [21] (Lemma 4.1) shows that, given V Z ⊆ X such that { V i , V j } ⊆ V Z , then

Equation (3) resulting from faithfulness and the directed global Markov property immediately implies that this expression can be used as a test of conditional independence to identify the Markov equivalence class of the graph M G V , as commonly done in constraint-based causal discovery (for reference, see e.g. Section 3 in Glymour et al. [4]). This result generalizes Lemma 1 of Montagna et al. [16], where it is used to define constraints to infer edges in the causal structure without latent variables.

Proposition 1 (Adapted 1 from [21]) . Let V be a set of random variables with strictly positive density generated according to model (4) . For each set V Z ⊆ V of nodes in M G V such that { V i , V j } ⊆ V Z , the following holds for each supported value v Z :

The result of Proposition 1 presents an alternative to conditional independence testing in constraint152 based approaches to causal discovery, showing that the equivalence class of the graph M G V can be 153 identified using the cross partial derivatives of the log-likelihood as a test of conditional independence 154 between variables, much in the spirit of the Fast Causal Inference algorithm [11]. Identifying the 155

Markov equivalence class is the most we can hope to achieve without further hypotheses. As we will 156 see in the next section, the score function can also help leverage additional restrictive assumptions on 157 the causal mechanisms of Equation (4) to identify direct causal effects. 158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

4 A theory of identifiability from the score

In this section, we show that, under additional assumptions on the data-generating process, we can identify the direct causal relations that are not influenced by unobserved variables, as well as the presence of unobserved active paths (Definition 5) between nodes in the marginalized graph M G V .

As a preliminary step before diving into causal discovery with latent variables, we show how the properties of the score function identify edges in directed acyclic graphs, that is in the absence of latent variables (when U = ∅ and G = M G V ). The goal of the next section is two-sided: first, it introduces the fundamental ideas connecting the score function to causal discovery that also apply to hidden variable models, second, it extends the existing theory of causal discovery with score matching to additive noise models with both linear and nonlinear mechanisms.

4.1 Warm up: identifiability without latent confounders

In this section, we summarise and extend the theoretical findings presented in Montagna et al. [17], where the authors show how to derive constraints on the score function that identify the causal order of the DAG G where all the variables in the set X are observed. Define the structural relations of (1) as:

with three times continuously differentiable mechanisms h i , noise terms centered at zero, and strictly positive density p X . Given the Markov factorization of Equation (2), the components of the score function ∇ log p ( x ) are:

where CH G i denotes the set of children of node X i . We observe that if a node X s is a sink , i.e. a node satisfying CH G s = ∅ , then the summation over the children vanishes, implying that:

The key point is that the score component of a sink node is a function of its structural equation noise term, such that one could learn a consistent estimator of ∂ X s log p X from a set of observations of the noise term N s . Given that, in general, one has access to X samples rather than observations of the noise random variables, authors in Montagna et al. [17] show that N s of a sink node can be consistently estimated from i.i.d. realizations of X . For each node X 1 , . . . , X k , we define the quantity:

where X \ X i are the random variables in the set X \ { X i } . E [ X i | X \ X i ] is the optimal least squares predictor of X i from all the remaining nodes in the graph, and R i is the regression residual. For a sink node X s , the residual satisfies:

which can be seen by rewriting E [ X s | X \ X s ] = h s ( X PA G s ) + E [ N s | X DE G s , X ND G s ] = h s ( X PA G s ) + E [ N s ] , where X DE G s and X ND G s denotes the descendants and non-descendants of X s , respectively. Equations (9) and (11) together imply that the score ∂ N s log p ( N s ) is a function of R s , such that it is possible to find a consistent approximator of the score of a sink from observations of R s .

Proposition 2 (Generalization of Lemma 1 in Montagna et al. [17]) . Let X be a set of random variables, generated by a restricted additive noise model (Definition 9) with structural equations (7) , and let X j ∈ X . Consider r j in the support of R j . Then:

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

Our result generalizes Lemma 1 in Montagna et al. [17], as they assume X generated by an identifiable additive noise model with nonlinear mechanisms. Instead, we remove the nonlinearity assumption and make the weaker hypothesis of a restricted additive noise model, which is provably identifiable [9], in the formal sense defined in the appendix (Definition 8). This result doesn't come as a surprise, given the previous findings of Ghoshal and Honorio [14] showing that the score infers linear non-Gaussian additive noise models: Proposition 2 provides a unifying and general theory for the identifiability of models with potentially mixed linear and nonlinear mechanisms.

Based on these insights, Montagna et al. [17] propose the NoGAM algorithm to exploit the condition in (12) for identifying the causal order of the graph: being E [ ∂ X i log p ( X ) | R i ] the optimal least squares estimator of the score of node X i from R i , a sink node is characterized as the argmin i E [ E [ ∂ X i log p ( X ) | R i ] -∂ X i log p ( X )] 2 , where in practice the residuals R i , the score components and the least squares estimators are replaced by their empirical counterparts. After a sink node is identified, it is removed from the graph and assigned a position in the order, and the procedure is iteratively repeated up to the source nodes. Being the score estimated by score matching techniques [23], we usually make reference to score matching-based causal discovery.

̸

In the next section, we show how we can generalize these results to identify direct causal effects between a pair of variables in the marginal MAG M G V when U = ∅

4.2 Identifiability in the presence of latent confounders

̸

We now introduce the last of our main theoretical results, that is: given a pair of nodes V i , V j that are adjacent in the graph M G V with U = ∅ , we can use the score function to identify the presence of a direct causal effect between V i and V j , or that of an active path that is influenced by unobserved variables. Given that the causal model of Equation (4) ensures identifiability only up to the equivalence class, we need additional restrictive assumptions. In particular, we enforce an additive noise model with respect to both the observed and unobserved noise variables. This corresponds to an additive noise model on the observed variables with the noise terms recentered by the latent causal effects.

Assumption 1 (SCM assumptions) . The set of structural equations of the observable variables specified in (4) is now defined as:

assuming the mechanisms f i to be of class C 3 ( R | V PA G i | ) , and mutually independent noise terms with strictly positive density function. The N i 's are assumed to be non-Gaussian when f i is linear in some of its arguments.

Crucially, our hypothesis is weaker than those required by two state-of-the-art approaches, CAM-UV [13] and RCD [12]: CAM-UV assumes a Causal Additive Model (CAM) with structural equations with nonlinear mechanisms in the form V i := ∑ k ∈ PA G i f ik ( V k ) + ∑ U i k g ik ( U i k ) + N i , and RCD requires an additive noise model with linear effects of both the latent and observed causes. Thus, our model encompasses and extends the nonlinear and linear settings of CAM-UV and RCD, such that the theory developed in the remainder of the section is valid for a broader class of causal models.

Our first step is rewriting the structural relations in (13) as:

which provides an additive noise model in the form of (7). Next, we define the following regression residuals for any node V k in the graph M G V :

where V Z \{ k } denotes the set of random variables V Z \ { V k } .

Given these definitions, we are ready to show how directed edges, and the presence of unobserved variables can be identified from the score of linear and nonlinear additive noise models.

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

4.2.1 Identifiability of directed edges

Consider V i , V j adjacent nodes in the PAG P M G V : we want to investigate when a direct causal effect V i ∈ V PA G j can be identified from the score. We make the following observations: for V Z = V PA G j ∪ { V j } and V PA G j | = G d U j , by Equation (15) it follows

where we use V PA G j | = G d U j to write E [ ˜ N j | V Z \{ j } ] = E [ ˜ N j ] . Moreover, we note that V j is a sink node relative to M G V Z , the marginalization of G onto V Z . In analogy to the case without latent variables, we can show that ∂ V j log p ( V Z ) is a function of ˜ N j , the error term in the additive noise model of Equation (14), such that the score of V j can be consistently predicted from observations of the residual R j ( V Z ) .

Proposition 3. Let X be generated by a restricted additive noise model with structural equations (7) , and causal graph G . Consider V i , V j adjacent in M G V , marginalization of G . Further, assume that the score component ∂ V j log p ( V Z ) is not constant for uncountable values of V Z .

(ii) Let V Z ⊆ V , such that { V i , V j } ⊆ V Z . Then:

̸

̸

̸

Intuitively, the proposition has two essential implications. Part (i) provides the condition for the identifiability of the potential direct causal effect between a pair V i , V j , that is, when the association between V j and its observed parents is not influenced by active paths that involve latent variables. This condition is necessary: given an active path such that V PA G j | = d G U j , the score could not identify a direct causal effect V i → V j , which is the content of the second part of the proposition.

We have established theoretical guarantees of identifiability for linear and nonlinear additive noise models, even in the presence of hidden variables: we find that the score function is a means for the identifiability of all direct parental relations that are not influenced by unobserved variables; all the remaining arrowheads of the edges in the graph M G V are identified no better than in the equivalence class. Based on these insights, we propose AdaScore, a score matching-based algorithm for the inference of Markov equivalence classes, direct causal effects, and the presence of latent variables.

4.3 A score-based algorithm for causal discovery

Building on our theory, we propose AdaScore, a generalization of NoGAM to linear and nonlinear additive noise models with latent variables. The main strength of our approach is its adaptivity with respect to structural assumptions: based on the user's belief about the plausibility of several modeling assumptions on the data, AdaScore can output an equivalence class (using the condition of Proposition 1 instead of conditional independence testing in an FCI-like algorithm), a directed acyclic graph (as in NoGAM), or a mixed graph, accounting for the presence of unobserved variables. We now describe the version of our algorithm whose output is a mixed graph, where we rely on score matching estimation of the score and its Jacobian (Appendix C.2). At an intuitive level, we find unoriented edges using Proposition 1, i.e. checking for dependencies in the form of non-zero entries in the Jacobian of the score via hypothesis testing on the mean, and find the edges' directions via the condition of Proposition 3, i.e. by estimating residuals of each node X i and checking whether they can correctly predict the i -th score entry (the vanishing mean squared errors are verified by hypothesis test of zero mean). It would be tempting to simply find the skeleton (i.e. the graphical representation of the constraints of an equivalence class) first via the well-known adjacency search of the FCI algorithm and then iterate through all neighborhoods of all nodes to orient edges using Proposition 3. This would be prohibitively expensive, as finding the skeleton is well-known to have super-exponential computational complexity [11]. Instead, we propose an alternative solution: exploiting the fact that some nodes may not be influenced by latent variables, we first use Proposition 2 to find sink nodes

that are not affected by latents (using hypothesis testing to find vanishing mean squared error in the 278 score predictions from the residuals), in the spirit of the NoGAM algorithm. If there is such a sink, 279 we search all its adjacent nodes via Proposition 1 (plus an optional pruning step for better accuracy, 280 Appendix C.2), and orient the inferred edges towards the sink. Else, if no sink can be found, we pick 281 a node in the graph and find its neighbors by Proposition 1, orienting its edges using the condition in 282 Proposition 3 (score estimation by residuals under latent effects). This way, we get an algorithm that 283 is polynomial in the best case (Appendix C.3). Details on AdaScore are provided in Appendix C, 284 while a pseudo-code summary is provided in the Algorithm 1 box. 285

5 Experiments

We use the causally 2 Python library [26] to generate synthetic data with known ground truths, created as Erdös-Rényi sparse and dense graphs, respectively with probability of edge between pair of nodes equals 0 . 3 and 0 . 5 . We sample the data according to linear and nonlinear mechanisms with additive noise, where the nonlinear functions are parametrized by a neural network with random weights, a common approach in the literature [18, 26, 27, 28, 29]. Noise terms are sampled from a uniform distribution in the [ -2 , 2] range. Hidden causal effects are obtained by randomly picking two nodes and dropping the corresponding column from the data matrix. See Appendix D.1 for further details on the data generation. As metric, we consider the structural Hamming distance (SHD) [30, 31], a simple count of the number of incorrect edges, where missing and wrongly directed edges count as one error. We fix the level of the hypothesis tests of AdaScore to 0 . 05 , which is a common choice in the absence of prior knowledge. We compare AdaScore to NoGAM, CAM-UV, RCD, and DirectLiNGAM, whose assumptions are detailed in Table 1. In the main manuscript, we comment on the results on datasets of 1000 observations from dense graphs, with and without latent variables. Additional experiments including those on sparse networks are presented in Appendix E. Our synthetic data are standardized by their empirical variance to remove shortcuts in the data [18, 32].

Discussion. Our experimental results on models without latent variables of Figure 1a show that when causal relations are linear, AdaScore can recover the causal graph with accuracy that is comparable with all the other benchmarks, with the exception of DirectLiNGAM. On nonlinear data AdaScore presents better performance than CAM-UV, RCD, and DirectLiNGAM while being comparable to NoGAM in accuracy. This is in line with our expectations: in the absence of finite sample errors and in the fully observable setting, NoGAM and AdaScore are indeed the same algorithms. When inferring under latent causal effects, Figure 1b, our method performs comparably to CAMUV and RCD on graphs up to seven nodes while slightly degrading on nine nodes. Additionally, AdaScore outperforms NoGAM in this setting, as we would expect according to our theory. Overall, we observe that our method is robust to a variety of structural assumptions, with accuracy that is often comparable and sometimes better than competitors (as in nonlinear observable settings). We remark that although AdaScore does not clearly outperform the other baselines, its broad theoretical guarantees of identifiability are not matched by any available method in the literature; this makes it an appealing option for inference in realistic scenarios that are hard to investigate with synthetic data, where the structural assumptions of the causal model underlying the observations are unknown.

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

adascore

camuv

nogam

rcd

lingam

3

5

7

9

number of nodes

0

5

10

15

20

25

30

35

shd

linear

3

5

7

9

number of nodes

0

5

10

15

20

25

30

35

shd

nonlinear

(a) Fully observable model

3

5

7

9

number of nodes

0

5

10

15

20

25

30

shd

linear

3

5

7

9

number of nodes

0

5

10

15

20

25

30

35

shd

nonlinear

(b) Latent variables model

6 Conclusion

The existing literature on causal discovery shows a connection between score matching and structure 318 learning in the context of nonlinear ANMs: in this paper, (i) we formalize and extend these results 319 to linear SCMs, and (ii) we show that the score retains information on the causal structure even in the 320 presence of unobserved variables. Additionally, while previous works posit the accent on finding the 321 causal order through the score, we study its potential to identify the Markov equivalence class with a 322 constraint-based strategy that does not explicitly require tests of conditional independence, as well as 323 to identify direct causal effects. Our theoretical insights result in AdaScore: unlike existing approaches 324 for the estimation of causal directions, our algorithm provides theoretical guarantees for a broad class 325 of identifiable models, namely linear and nonlinear, with additive noise, in the presence of latent 326 variables. Even though AdaScore does not clearly outperform the existing baselines on our synthetic 327 benchmark, its adaptivity to different structural hypotheses is a step towards causal discovery that is 328 less reliant on prior assumptions, which are often untestable and thus hindering reliable inference in 329 real-world problems. While we do not touch on the task of causal representation learning [33], where 330 causal variables are learned from data, we believe this is a promising research direction in relation 331 to our work due to the specific interplay between score-matching estimation and generative models. 332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

References

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Judea Pearl. Causality . Cambridge university press, 2009.

Peter Spirtes. Introduction to causal inference. Journal of Machine Learning Research , 11(54): 1643-1662, 2010. URL http://jmlr.org/papers/v11/spirtes10a.html .

Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in Genetics , 10, 2019. ISSN 1664-8021. doi: 10.3389/fgene.2019. 00524. URL https://www.frontiersin.org/articles/10.3389/fgene.2019.00524 .

P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search . MIT press, 2nd edition, 2000.

David Maxwell Chickering. Optimal structure identification with greedy search. J. Mach. Learn. Res. , 3(null):507-554, mar 2003. ISSN 1532-4435. doi: 10.1162/153244303321897717. URL https://doi.org/10.1162/153244303321897717 .

Shohei Shimizu, Patrik O. Hoyer, Aapo Hyvärinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. J. Mach. Learn. Res. , 7:2003-2030, dec 2006. ISSN 1532-4435.

Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/paper/2008/file/ f7664060cc52bc6f3d620bcedc94a4b6-Paper.pdf .

Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. J. Mach. Learn. Res. , 15(1):2009-2053, jan 2014. ISSN 1532-4435.

Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , UAI '09, page 647-655, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 9780974903958.

Peter Spirtes. An anytime algorithm for causal inference. In Thomas S. Richardson and Tommi S. Jaakkola, editors, Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics , volume R3 of Proceedings of Machine Learning Research , pages 278-285. PMLR, 04-07 Jan 2001. URL https://proceedings.mlr.press/r3/spirtes01a.html . Reissued by PMLR on 31 March 2021.

Takashi Nicholas Maeda and Shohei Shimizu. Rcd: Repetitive causal discovery of linear non-gaussian acyclic models with latent confounders. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 735-745. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/v108/maeda20a.html .

Takashi Nicholas Maeda and Shohei Shimizu. Causal additive models with unobserved variables. In Uncertainty in Artificial Intelligence , pages 97-106. PMLR, 2021.

Asish Ghoshal and Jean Honorio. Learning linear structural equation models in polynomial time and sample complexity. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1466-1475. PMLR, 09-11 Apr 2018. URL https://proceedings.mlr.press/v84/ghoshal18a.html .

Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear additive noise models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on

Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18741382 18753. PMLR, 17-23 Jul 2022. 383

384

385

386

Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Scalable causal discovery with score matching. In 2nd Conference on Causal Learning and Reasoning , 2023. URL https://openreview.net/forum?id=6VvoDjLBPQV .

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Causal discovery with score matching on additive models with arbitrary noise. In 2nd Conference on Causal Learning and Reasoning , 2023. URL https://openreview.net/forum?id= rVO0Bx90deu .

Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, and Francesco Locatello. Shortcuts for causal discovery of nonlinear models by score matching, 2023.

Pedro Sanchez, Xiao Liu, Alison Q O'Neil, and Sotirios A. Tsaftaris. Diffusion models for causal discovery via topological ordering. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Idusfje4-Wq .

Zhenyu Zhu, Francesco Locatello, and Volkan Cevher. Sample complexity bounds for scorematching: Causal discovery and generative modeling. Advances in Neural Information Processing Systems , 36, 2024.

Alessio Spantini, Daniele Bigoni, and Youssef Marzouk. Inference via low-dimensional couplings, 2018.

Juan Lin. Factorizing multivariate function classes. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems , volume 10. MIT Press, 1997. URL https://proceedings.neurips.cc/paper_files/paper/1997/ file/8fb21ee7a2207526da55a679f0332de2-Paper.pdf .

Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res. , 6:695-709, 2005. URL https://api.semanticscholar.org/CorpusID: 1152227 .

Caroline Uhler, G. Raskutti, Peter Bühlmann, and B. Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics , 41, 07 2012. doi: 10.1214/12-AOS1080.

Jiji Zhang. Causal reasoning with ancestral graphs. Journal of Machine Learning Research , 9 (7), 2008.

Francesco Montagna, Atalanti Mastakouri, Elias Eulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janzing, Bryon Aragam, and Francesco Locatello. Assumption violations in causal discovery and the robustness of score matching. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 47339-47378. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 93ed74938a54a73b5e4c52bbaf42ca8e-Paper-Conference.pdf .

Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclicity constraints. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=eYciPrLuUhG .

Nan Rosemary Ke, Silvia Chiappa, Jane X Wang, Jorg Bornschein, Anirudh Goyal, Melanie Rey, Theophane Weber, Matthew Botvinick, Michael Curtis Mozer, and Danilo Jimenez Rezende. Learning to induce causal structure. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=hp_RwhKDJ5 .

Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning , 65:31-78, 2006.

Sofia Triantafillou and Ioannis Tsamardinos. Score-based vs constraint-based causal learning in the presence of confounders. In Cfa@ uai , pages 59-67, 2016.

Alexander G. Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! causal discovery benchmarks may be easy to game. In Neural Information Processing Systems , 2021. URL https://api.semanticscholar.org/CorpusID:239998404 .

Bernhard Scholkopf, Francesco Locatello, Stefan Bauer, Nan Ke, Nal Kalchbrenner, Anirudh Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE , PP: 1-23, 02 2021. doi: 10.1109/JPROC.2021.3058954.

Peter Spirtes and Thomas Richardson. A polynomial time algorithm for determining dag equivalence in the presence of latent variables and selection bias. In Proceedings of the 6th International Workshop on Artificial Intelligence and Statistics , pages 489-500. Citeseer, 1996.

Yingzhen Li and Richard E Turner. Gradient estimators for implicit models. arXiv preprint arXiv:1705.07107 , 2017.

Peter Bühlmann, Jonas Peters, and Jan Ernest. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics , 42(6), dec 2014. URL https://doi.org/10.1214%2F14-aos1260 .

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

A Useful results

In this section, we provide a collection of results and definitions relevant to the theory of this paper.

A.1 Definitions over graphs

Let X = X 1 , . . . , X d a set of random variables. A graph G = ( X,E ) consists of finitely many nodes or vertices X and edges E . We now provide additional definitions, separately for directed acyclic and mixed graphs.

Directed acyclic graph. In a directed graph , nodes can be connected by a directed edge ( → ), and between each pair of nodes there is at most one directed edge. We say that X 1 is a parent of X j if X i → X j ∈ E , in which case we also say that X j is a child of X i . Two nodes are adjacent if they are connected by an edge. Three nodes are called a v-structure if one node is a child of the other two, e.g. as X i → X k ← X j is a collider. A path in G is a sequence of at least two distinct vertices X i 1 , . . . , X i m such that there is an edge between X i k and X i k +1 . If X i k → X i k +1 for every node in the path, we speak of a directed path , and call X i k an ancestor of X i k +1 , X i k +1 a descendant of X i k . Given the set DE G i of descendants of a node X i , we define the set of non-descendants of X i as ND G i = X \ (DE G i ∪{ X i } ) . Anode without parents is called a source node . Anode without children is called a sink node . A directed acyclic graph is a directed graph with no cycles.

Mixed graph. In a mixed graph nodes can be connected by a directed edge ( → ) or a bidirected edge ( ↔ ), and between each pair of nodes there is at most one directed edge. Two vertices are said to be adjacent in a graph if there is an edge (of any kind) between them. The definitions of parent , child , ancestor , descendant , path provided for directed acyclic graph also apply in the case of mixed graphs. Additionally, X i is a spouse of X j (and vice-versa) if X i ↔ X j ∈ E . An almost directed cycle occurs when X i ↔ X j ∈ E and X i is an ancestor of X j in G .

For ease of reference from the main text, we separately provide the definition of inducing paths and ancestors in directed acyclic graphs.

Definition 2 (Ancestor) . Consider a DAG G with set of nodes X , and X i , X j elements of X . We say that X i is an ancestor of X j if there is a directed path from X i to X j in the graph, as in X i → . . . → X j .

Definition 3 (Inducing path) . Consider a DAG G with set of nodes X , and Y, Z disjoint subsets such that X = Y ˙ ∪ Z . We say that there is an inducing path relative to Z between the nodes Y i , Y j if every node on the path that is not in Z ∪{ Y i , Y j } is a collider on the path (i.e. for each Y k ∈ Y on the path the sequence Y i . . . → Y k ← . . . Y j appears) and every collider on the path is an ancestor of Y i or Y j .

One natural way to encode inducing paths and ancestral relationships between variables is represented by maximal ancestral graphs.

Definition 4 (MAG) . A maximal ancestral graph (MAG) is a mixed graph such that:

there are no directed cycles and no almost directed cycles;

there are no inducing paths between two non-adjacent nodes.

Next, we define conditional independence in the context of graphs.

Definition 5 (m-separation) . Let M be a mixed graph with nodes X . Apath π in M between X i , X j elements of X is active w.r.t. Z ⊆ X \ { X i , X j } if:

every non-collider on π is not in Z

every collider on π is an ancestors of a node in Z .

X i and X j are said to be m-separated by Z if there is no active path between X i and X j relative to Z . Two disjoint sets of variables W and Y are m-separated by Z if every variable in W is m-separated from every variable in Y by Z .

If m-separation is applied to DAGs, it is called d-separation .

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

The set of directed acyclic graphs that satisfy the same set of conditional independencies form an equivalence class, known as the Markov equivalence class .

Definition 6 (Markov equivalence class of a DAG) . Let G be a DAG with nodes X . We denote with [ G ] the Markov equivalence class of G . A DAG ˜ G with nodes X is in [ G ] if the following conditions are satisfied for each pair X i , X j of distinct nodes in X :

there is an edge between X i , X j in G if and only if there is an edge between X i , X j in ˜ G ;

let π be a path between X i and X j . X k is a collider in the path π in G if and only if it is a collider in the path π in ˜ G .

In summary, graphs in the same equivalence class share the edges up to direction, the set of dseparations, and the set of colliders.

Just as for DAGs, there may be several MAGs that imply the same conditional independence statements. Denote the Markov-equivalence class of a MAG M with [ M ] : this is represented by a partial mixed graph, the class of graphs that can contain four kinds of edges: → , ↔ , ◦- -◦ and ◦→ , and hence three kinds of end marks for edges: arrowhead ( > ), tail ( -) and circle ( ◦ ).

Definition 7 (PAG, Definition 3 of Zhang [25]) . Let [ M ] be the Markov equivalence class of an arbitrary MAG M . The partial ancestral graph (PAG) for [ M ] , P M , is a partial mixed graph such that:

P M has the same adjacencies as M (and any member of [ M ] ) does;

A mark of arrowhead is in P M if and only if it is shared by all MAGs in [ M ] ; and

A mark of tail is in P M if and only if it is shared by all MAGs in [ M ] .

Intuitively, a PAG represents an equivalence class of MAGs by displaying all common edge marks shared by all members of the class and displaying circles for those marks that are not in common.

A.2 Equivalence between m-separation and d-separation

In this section, we provide a proof for equation (5), stating the equivalence between m-separation and d-separation in a formal sense.

Lemma 1 (Adapted from Zhang [25]) . Let G be a DAG with nodes X = V ∪ U , with V and U disjoint sets, and M G V the marginalization of G onto V . For any { V i , V j } ∈ V and V Z ⊆ V \{ V i , V j } , the following equivalence holds:

Proof. The implication V i | = d G V j | V Z = ⇒ V i | = m M G V V j | V Z is a direct consequence of Lemma 18 from Spirtes and Richardson [34], where we set S = ∅ , since we do not consider selection bias. The implication V i | = d G V j | V Z ⇐ = V i | = m M G V V j | V Z follows from Lemma 17 by Spirtes and Richardson [34], again with S = ∅ . Note, that in their terminology 'd-separation in MAGs' is what we call m-separation.

A.3 Additive noise model identifiability

We study the identifiability of the additive noise model, reporting results from Peters et al. [9]. We start with a formal definition of identifiability in the context of causal discovery.

Definition 8 (Identifiable causal model) . Let ( X,N, F , p N ) be an SCM with underlying graph G and p X joint density function of the variables of X . We say that the model is identifiable from observational data if the distribution p X can not be generated by a structural causal model with graph ˜ G ̸ = G .

First, we consider the case of models of two random variables

531

532

533

534

535

536

537

538

539

540

541

for all pairs ( x 1 , x 2 ) . 542

Proof. By contradiction, assume that there exists ( x 1 , x 2 ) such that ∂ ∂X 1 ( ξ ′ ( x 1 ) -f ′ ( x 1 ) ν ′ ( x 2 -f ( x 1 ))) = 0 . Then:

where π ( x 1 , x 2 ) = log p ( x 1 , x 2 ) . By explicitly computing all the partial derivatives of the above equation, we obtain that equation 18 is satisfied, which violates Condition 1.

These results guaranteeing the identifiability of the bivariate additive noise model can be generalized to the multivariable case, with a set of random variables X = { X 1 , . . . , X k } that satisfy:

where G is the resulting causal graph directed and acyclic. The intuition is that, rather than studying the multivariate model as a whole, we need to ensure that Condition 1 is satisfied for each pair of nodes, adding restrictions on their marginal conditional distribution.

Definition 9 (Definition 27 of Peters et al. [9]) . Consider an additive noise model with structural equations (19). We call this SCM a restricted additive noise model if for all X j ∈ X , X i ∈ X PA G j , and all sets X S ⊆ X , S ⊂ N , with X PA G j \ { X i } ⊆ X S ⊆ X G ND j \ { X i , X j } , there is a value x S with p ( x S ) > 0 , such that the triplet

satisfies Condition 1. Here, f j ( x PA G j \{ i } , · ) denotes the mechanism function x i ↦→ f j ( x PA G j ) . Additionally, we require the noise variables to have positive densities and the functions f j to be continuous and three times continuously differentiable.

Then, for a restricted additive noise model, we can identify the graph from the distribution.

̸

̸

Theorem 2 (Theorem 28 of Peters et al. [9]) . Let X be generated by a restricted additive noise model with graph G , and assume that the causal mechanisms f j are not constant in any of the input arguments, i.e. for X i ∈ X PA G j , there exist x i = x ′ i such that f j ( x PA G j \{ i } , x i ) = f j ( x PA G j \{ i } , x ′ i ) .

Then,

G

is identifiable.

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

̸

Condition 1 (Condition 19 of Peters et al. [9]) . Consider an additive noise model with structural equations (17). The triple ( f, p X 1 , p N ) does not solve the following differential equation for all pairs x 1 , x 2 with f ′ ( x 2 ) ν ′′ ( x 2 -f ( x 1 )) = 0 :

Here, ξ := log p X 1 , ν := log p N , the logarithms of the strictly positive densities. The arguments x 2 -f ( x 1 ) , x 1 , and x 1 of ν , ξ and f respectively, have been removed to improve readability.

Next, we show that a structural causal model satisfying Condition 1 is identifiable, as in Definition 8

Theorem 1 (Theorem 20 of Peters et al. [9]) . Let p X 1 ,X 2 the joint distribution of a pair of random variables generated according to the model of equation (17) that satisfies Condition 1, with graph G . Then, G is identifiable from the joint distribution.

Finally, we show an important fact, holding for identifiable bivariate models, which is that the score ∂ ∂X 1 log p ( x 1 , x 2 ) is nonlinear in x 1 .

Lemma 2 (Sufficient variability of the score) . Let p X 1 ,X 2 the joint distribution of a pair of random variables generated according to a structural causal model that satisfies Condition 1, with graph G . Then:

̸

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

A.4 Other auxiliary results

Westate several results that hold for a pair of random variables that are not connected by an active path that includes unobserved variables (active paths are introduced in Definition 5). For the remainder of the section, let V, U be a pair of disjoint sets of random variables, X = V ∪ U generated according to the structural causal model defined by the set of equations (1), G the associated causal graph, and M G V the marginalization onto V .

The first statement provides under which condition the unobserved parents of two variables in the marginal MAG are mutually independent random vectors.

̸

Lemma 3. Let V j ∈ V , and Z ⊂ N such that V Z = V PA G j ∪ { V j } . Assume V PA G j | = d G U j . Then U j | = d G U Z k for each index Z k = j .

̸

Proof. The assumption V PA G j | = d G U j implies that there is no active path in G between nodes in V PA G j and nodes in U j . Given that for each Z k ∈ Z , Z k = Z , nodes in U Z k are direct causes of at least one node in V PA G j , any active path between nodes in U Z k and nodes in U j would also be an active path between V PA G j and U j , which is a contradiction. Hence U j | = d G U Z k .

The previous lemmas allow proving the following result, which will be fundamental to demonstrate the theory of Proposition 3.

Lemma 4. Let V j ∈ V , and Z ⊂ N such that V Z = V PA G j ∪ { V j } . Assume V PA G j | = d G U j . W.l.o.g., let the j -th element of V Z be V Z j = V j . Denote as U Z the set of unobserved parents of nodes in V Z , and U Z \{ j } the unobserved parents of nodes in V Z \{ j } := V Z \ V j . Then, the following holds for each v Z , u Z values:

̸

Proof. By the law of total probability and the chain rule, we can write p ( v Z ) as: 574

̸

By Lemma 3, U Z j | = U Z k , k = j , where U Z k denotes unobserved parents of the node V Z k . Then, 575 we can factorize p ( u ) = p ( u Z j ) p ( u Z \{ j } ) . Plugging the factorization in equation (20) we find 576

̸

where the latter equation comes from the global Markov property on the graph G . Further, by assump577 tion of V PA G j | = d G U j , we know that U Z j | = V Z k , k = j , such that p ( v Z \{ j } | u ) = p ( v Z \{ j } | u Z \{ j } ) . 578 Then: 579

580

which proves the claim.

where

581

582

583

584

585

586

Intuitively, Lemma 4 shows that given a node V j without children and bidirected edges in a marginalized graph M G V Z , the kernel of node V j in the Markov factorization of p ( v Z ) is equal to the kernel of the same node in the Markov factorization of p ( x ) of equation (2), relative to the graph without latent confounders G .

B Proofs of theoretical results

B.1 Proof of Proposition 1

Proof of Proposition 1. Observe that

where the first equivalence holds by a combination of the faithfulness assumption with the global 587 Markov property, as explicit in equation (3), and the second due to Lemma 1. Then, the claim is 588 proven. 589

B.2 Proof of Proposition 2 590

Proof. The forward direction is immediate from equation (9) and R j = N j , when X j is a sink (equation (11)). Thus, we focus on the backward direction. Given

we want to show that X j has no children, which we prove by contradiction. 591

Let us introduce a function q : R → R such that:

The mean squared error equal to zero implies that s j ( X ) is a constant, once R j is observed. Formally, under the assumption of p ( x ) > 0 for each x ∈ R k , this implies that

̸

By contradiction, we assume that X j is not a leaf, and want to show that s j ( X ) is not constant in X , given R j fixed. Let X i such that X j ∈ X PA G i . Being the structural causal model identifiable, there is no model with distribution p X whose graph has a backward edge X i → X j : thus, the Markov factorization of equation (2) is unique and implies:

We note that, by definition of residual in equation (10), R j = r j fixes the following distance:

Hence, conditioning on R j doesn't restrict the support of X : given R j = r j , for any x \ X j (value of the vector of elements in X \ { X j } ), ∃ n j with p ( n j > 0) (by the hypothesis of strictly positive densities of the noise terms) that satisfies

Next, we condition on all the parents of X i , except for X j , to reduce our problem to the simpler 592 bivariate case. Let S ⊂ N and X S ⊆ X such that X PA G i \ { X j } ⊆ X S ⊆ X ND G i \ { X i , X j } , 593 and consider x S such that p ( x S > 0) . Let X PA G i = x PA G i hold under X S = x S . We define 594 X j | x s := X j | ( X S = x S ) , and similarly X | x s := X | ( X S = x S ) . Being the SCM a restricted 595

and s j : R | X | → R ,

601

602

603

604

605

606

607

608

609

610

additive noise model, by Definition 9, the triplet ( g i , p X j | x s , p N i ) satisfies Condition 1, where 596 g i ( x j ) = h i ( x PA G i \{ X j } , x j ) . Consider X i = x i , and the pair of values ( x j , x ∗ j ) such that x j = x ∗ j 597 and 598

̸

̸

where we resort to the usual notation ν N i := log p N i . By Lemma 2, ( x i , x j ) and ( x i , x ∗ j ) satisfy: 599

̸

̸

where ξ := log p X j | x s . Thus, we can fix x j and x ∗ j (which are arbitrarily chosen) such that 600

̸

Fixing X | x S, x j = x and X | x S, x ∗ j = x ∗ , where the two values differ only in their jth component, we find the following difference:

which is different from 0 by equation (21). This contradicts the fact that the score s j is constant once R j is fixed, which proves our claim.

B.3 Proof of Proposition 3

In this proof, we use several ideas from the demonstration of Proposition 2. We demonstrate the forward and the backward parts of the two statements separately.

Proof of part (i), forward direction. Given V Z = V PA G j ∪ { V i , V j } and r j ∈ R in the image of R j , we want to show:

By Lemma 4, the score of V j is

for some Q map acting on V Z . The latter equality holds because all variables in V Z are nondescendants of V j , such that ∂ V j Q ( V Z ) = 0 . Further, by equation (16) we know that

where c = -E [ ˜ N j ] is a constant. It follows that the least square estimator of the score of V j from R j ( V Z ) satisfies the following equation:

where the first equality holds because E [ ·| ˜ N j ] = E [ ·| ˜ N j + c ] . Then, we find

Proof of part (i), backward direction. Given V Z = V PA G j ∪{ V i , V j } , r j ∈ R in the image of R j , and

we want to show that V PA G j | = d G U j ∧ V i ∈ V PA G j , meaning that there is a direct causal effect that 611 is not biased by unobserved variables. We provide the proof by contradiction, in analogy to the 612 demonstration of the backward direction of Proposition 2. 613

̸

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

Let us introduce s j : R | V Z | → R , and further define:

̸

The mean squared error equal to zero implies that s j ( V Z ) is constant in V Z , once R j is observed. By contradiction, we assume that V PA G j | = d G U j ∨ V i ̸∈ V PA G j , and want to show that s j ( V Z ) is not constant in V Z , given R j fixed. In this regard, we make the following observation: by definition of residual in equation (15), R i ( V Z ) = r i fixes the following distance:

Hence, conditioning on R j ( V Z ) doesn't restrict the support of V Z : given R j ( V Z ) = r j , ∃ ˜ n j with p (˜ n j ) > 0 (by assumption of strictly positive densities p N j and p X ), that satisfies

̸

for all v Z \{ j } . Hence, the random variable V Z | R j ( V Z ) = r j has strictly positive density on all points v Z where p V Z ( v Z ) > 0 . Now, consider v Z and v ∗ Z , taken from the set of uncountable values such that the score s j function is not a constant, meaning that s j ( v Z ) = s j ( v ∗ Z ) , where V Z is sampled given R j ( V Z ) = r j . Given that different v Z and v ∗ Z are selected from an uncountable subset of the support, we conclude that the score s j | ( R j ( V Z ) = r j ) = ∂ V j log p ( V Z | R j ( V Z ) = r j ) is not a constant for at least an uncountable set of points, which contradicts equation (22).

̸

Proof of part (ii), forward direction. Given that V i is connected to V j in the marginal MAG and that V PA G j | = d G U j ∨ V i ̸∈ V PA G j , we want to show that for each V Z ⊆ V with { V i , V j } ⊆ V Z , the following holds:

̸

Let us introduce h : R → R such that:

Having the mean squared error in equation (23) equals zero implies that s j ( V Z ) is a constant, once R j ( V Z ) is observed. Thus, the goal of the proof is to show that there are values of V Z such that the score is not a constant once R j is fixed. By definition of residual in equation (15), R j ( V Z ) = r j fixes the following distance:

Hence, conditioning on R j ( V Z ) doesn't restrict the support of V Z : given R j ( V Z ) = r j , ∃ ˜ n j with p (˜ n j ) > 0 (by assumption of positive density of the noise N j on the support R ), that satisfies

̸

for all v Z \{ j } . Hence, the random variable V Z | R j ( V Z ) = r j has strictly positive density on all points v Z where p V Z ( v Z ) > 0 . Now, consider v Z and v ∗ Z , taken from the set of uncountable values such that the score s j function is not a constant, meaning that s j ( v Z ) = s j ( v ∗ Z ) , where V Z is sampled given R j ( V Z ) = r j . Given that different v Z and v ∗ Z are selected from an uncountable subset of the support, we conclude that the score s j | ( R j ( V Z ) = r j ) = ∂ V j log p ( V Z | R j ( V Z ) = r j ) is not a constant for at least an uncountable set of points, such that the claim follows.

̸

Proof of part (ii), backward direction. Given that E [ ∂ V j log p ( V Z ) -E [ ∂ V j log p ( V Z ) | R j ( V Z ) = r j ]] 2 = 0 for all V Z ⊆ V such that { V i , V j } ∈ V Z , and given V i and V j adjacent in the marginal MAG, we want to show that

̸

The prove comes easily by contradiction: say that V PA G j | = d G U j ∧ V i ∈ V PA G j . Then, by the forward 629 direction of part (i) of Proposition 3, we know that V Z = V PA G j ∪ { V j } satisfies E [ ∂ V j log p ( V Z ) -630 E [ ∂ V j log p ( V Z ) | R j ( V Z ) = r j ]] 2 = 0 , leading to a contradiction. 631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

C Algorithm

C.1 Detailed description of our algorithm

In Proposition 1 we have seen that score matching can detect m -separations and therefore the skeleton of the PAG describing the data. If one is willing to make the assumptions required for Proposition 3 it could be desirable to use this to orient edges, since the interpretation of PAG edges might be cumbersome for people not familiar with ancestral models. Therefore, one could simply find the skeleton of the PAG using the fast adjacency search [5] and then orient the edges by applying Proposition 3 on every subset of the neighbourhood of every node. This would yield a very costly algorithm. But if we make the assumptions required to orient edges with Proposition 3 we can do a bit better. In Algorithm 2 we present an algorithm that still has the same worst case runtime but runs polynomially in the best case. The main intuition is that we iteratively remove irrelevant nodes in the spirit of the original SCORE algorithm [15]. To this end, we first check if the is any unconfounded sink if we consider the set of all remaining variables. If there is one, we can orient its parents and ignore it afterwards. If there is no such set, we need to fall back to the procedure proposed above, i.e. we need to check the condition of Proposition 3 on all subsets of the neighbourhood of a node, until we find no node with a direct outgoing edge. In Proposition 4 we show that this way we do not fail orient edge or fail to remove any adjacency. In the following discussion, we will use the notation

for the second residual from Proposition 3 and also

for the cross-partial derivative, where X i , X j ∈ V and Z ⊆ V .

Proposition 4 (Correctness of algorithm) . Let X = V ˙ ∪ U be generated by the SCM in Equation (4) with non-constant scores for uncountably many values. Let G X be the causal DAG of X and G V be the marginal MAG of G X . Then Algorithm 2 outputs a directed edge from X i ∈ V to X j ∈ V iff there is a direct edge in G X between them and no unobserved backdoor path w.r.t. U . Further, the output of Algorithm 2 has the same skeleton as G V .

Proof. We proof the statement by induction over the steps of the algorithm. Let S be the set of remaining nodes in an arbitrary step of the algorithm. Our induction hypothesis is that for X i , X j ∈ S and X k ∈ B i we have

X i is an unconfounded sink w.r.t. to some set S ′ ⊆ S iff X i is an unconfounded sink w.r.t. some S ′′ ⊆ V

if there is no S ′ ⊆ V \ { X i , X j } such that X i | = X j | S ′ then X j ∈ B i

Clearly, this holds in the initial step as S = V .

Suppose we find δ i ( X S ) = 0 for X i ∈ S . If X i has at least one adjacent node in M G V , by Proposition 3, we know that X i does not have any children and is also not connected to any other node in S via a hidden mediator or unobserved confounder. This means, all nodes that are not separable from X i must be direct parents of X i , which are by our induction hypothesis 2) the nodes in B i . Since X i does not have children, it also suffices to check X i | = X j | S \ { X i , X j } for X j ∈ B i (instead of conditioning on all subsets of B i ). So we can already add these direct edges to the output. If, on the other hand, X i has no adjacent nodes in M G V , we have X i | = X j | S \{ X i , X j } for X j ∈ B i , so in both cases we add the correct set of parents. Since X i is not an ancestor of any of the nodes in S \ { X i } , X i cannot be a hidden mediator or hidden confounder between nodes in S \ { X i } and conditioning on X i cannot block an open path. Thus, the induction hypothesis still holds in the next step.

Suppose now there is no unconfounded sink and we explore X i . By our induction hypothesis 2), B i contains the parents of X i and by Proposition 3 it suffices to only look at subsets of B i to orient direct edges. And also due to the induction hypothesis 2) B i contains all nodes that are not separable from X i . So by adding bidirected edges to all nodes in B i can only add too many edges but not miss some.

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

̸

̸

̸

Now it remains to show that the induction hypothesis holds if we set S to S \ { X i } . For 1) we need to show that X i cannot be a hidden mediator or hidden confounder w.r.t. S \ { X i } (since ignoring X i won't change whether there is a direct edge or not). Suppose X i is on a unobserved causal path X k →··· → U m → X l with X k , X l ∈ S \ { X i } and U m ∈ X \ ( S \ { X i } ) . This path must have been a unobserved causal path before, unless X i = U m . But then there is a direct edge X i → X l . We would not remove X i from S if this edge was unconfounded, so there must a hidden confounder between X i and X l . But in this case, Proposition 3 wouldn't allow us to direct the edge anyway, since V PA l | = d G U l . Suppose there is confounding path X k ←··· → U m → X l with X k , X l ∈ S \ { X i } and U m ∈ X \ ( S \ { X i } ) . If X i = U m the path was already been a confounding path without X i being unobserved. So again, there must be a confounder between X i and X l , as otherwise we would not remove X i . And analogously to before, we could not have oriented the edge even with X i ∈ S since V PA l | = d G U l . For 2) we only have to see that we just remove nodes from B i if we found an independence.

For | S | < 2 , the algorithm enters the final pruning stage. From the discussion above it is clear, that we already have the correct result, up to potentially too many bidirected edges. In the final step we certainly remove all these edges X i ↔ X j , as we check m -separation for all subsets of the neighbourhoods Adj( X i ) and Adj( X j ) , which are supersets of the true neighbourhoods.

̸

̸

̸

̸

696

C.2 Finite sample version of AdaScore

All theoretical results in the paper have assumed that we know the density of our data. Obviously, in 697 practise we have to deal with a finite sample instead. Especially, in Proposition 1 and Proposition 3 698 we derived criteria that compare random variables with zero. Clearly, this condition is never met in 699 practise. Therefore, we need find ways to reasonably set thresholds for these random quantities. 700

First note, that we use the Stein gradient estimator [35] to estimate the score function. This means 701 especially that for a node V i we get a vector 702

i.e. an estimate of the score for every one of the m samples. Analogously, we get a m × d × d tensor 703 for the estimates of ∂ 2 ∂V i ∂V j log p ( v ) . 704

In Proposition 1 we showed that 705

In the finite sample version, we use a one sample t-test on the vector of estimated cross-partial 706 derivatives with the null-hypothesis that the means is zero. Due to the central limit theorem, the 707 sample mean follows approximately a Gaussian distribution, regardless of the true distribution of the 708 observations. 709

710

711

712

713

714

715

716

For Proposition 3 we need to do some additional steps. Recall, that the relevant quantity in Proposition 3 is the mean squared error of a regression, which is always positive. Therefore, a test for mean zero is highly likely to reject in any case. We decided to employ a two-sample test in a similar (but different) manner as Montagna et al. [17]. As test, we used the Mann-Whitney U-test. Note, that Algorithm 2 employs Proposition 3 in two different ways: first, to decide whether there is an unconfounded sink and second, to orient edges in case there is no unconfounded sink. We pick a different sample as second sample of the Mann-Whitney U-test.

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

Analogously to before, this is a vector with m entries, one for every sample.

Note, that in the case where we want to check if there is an unconfounded sink, we do not make any mistake by rejecting too few hypotheses, i.e. if we miss some unconfounded sinks (instead, we only lose efficiency, as we do the costly iteration over all possible sets of parents). Therefore, for this test we chose a a second sample that yields a 'conservative' test result.

̸

As candidate sink for set S ⊆ V , we pick the node X i = min i mean( δ i ( X S )) . In fact, we want to know whether the mean of δ i is significantly lower than all other means. But we empirically observed that choosing the concatenated δ s of all nodes as second sample makes the test reject with very high probability, which would lead our algorithm to falsely assume the existence of an unconfoudned sink. Instead, we then pick as second 'reference node' X j = min j = i mean( δ j ( X Z )) . We then do the two sample test between δ i ( X Z ) and δ j ( X Z ) . The intuition is that the test will reject the hypothesis of identical means, if X i is an unconfounded sink but X j is not.

In the case where we use Proposition 3 to orient edges, we only need to decide whether an not previsouly directed edge X i -X j needs to be oriented one way, the other way, or not at all. Instead, here the issue lies in the fact that we need to iterate over possible sets of parents of the nodes. Let B i be the set of nodes that have not been m -separated from X i by any test so far. We pick the subset Z i = min Z ′ ⊆ B i mean( δ Z ′ i ) , i.e. the set with the lowest mean error. We then conduct the test with δ i ( X Z i ) and δ j ( X Z j ) . If there is a directed edge between them, one of the residuals will be significantly lower than the other.

Just like Montagna et al. [17] we use a cross-validation scheme to generate the residuals, in order to prevent overfitting. We split the dataset into several equally sized, disjoint subsamples. For every residual we fit the regression on all subsamples that don't contain the respective target.

Also, just like in the NoGAM algorithm Montagna et al. [17] we add a pruning step for the directed 739 edges to the end. The idea is to use a feature selection method to remove insignificant edges. Just like 740 Montagna et al. [17], we use the CAM-based pruning step proposed by Bühlmann et al. [36], which 741 fits a generalised additive regression model from the parents to a child and test whether one of the 742

additive components is significantly non-zero. All parents for which the test rejects this hypothesis 743 are removed. 744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

C.3 Complexity

Proposition 5. Complexity Let n be the number of samples and d the number of observable nodes. Algorithm 2 runs in

where r ( n, d ) is the time required to solve a regression problem and s ( n, d ) is the time for calculating the score. With e.g. kernel-ridge regression and the Stein-estimator, both run in O ( n 3 ) .

Proof. Algorithm 2 runs its main loop d times. It first checks for the existence of an unconfounded sink, which involves solving 2 d regression problems (including cross-validation prediction) and calculating the score, adding up to (2 d 2 -d ) regressions and d score evaluations. In the worst case, we detect no unconfounded sink and iterate through all subsets of the neighbourhood of a node (which is in the worst case of size d -1 ) and for all other nodes in the neighbourhood we solve 2 d regression problems and evaluate the score. For each subset we calculate two regression functions, the score and calculate the entries in the Hessian of the log-density, i.e. d · 2 d regressions, d · 2 d -1 scores and additionally 2 d -1 Hessians. If we are unlucky, this node has a directed outgoing edge and we continue with this node (with the same size of nodes). This can happen d -1 times. So we get ( d 2 -d ) · 2 d regressions and ( d 2 -d ) · 2 d -1 scores and Hessians. In the final pruning step we calculate for every bidirected edge (of which there can be ( d 2 -d ) / 2 ) a Hessian for all subsets of the neighbourhoods, which can again be 2 d -1 subsets. Using the pruning procedure from CAM for the directed edges we also spend at most O ( nd 3 ) steps.

In the best case, we always find an unconfounded sink. Then our algorithm reduces to NoGAM.

D Experimental details

In this section, we present the details of our experiments in terms of synthetic data generation and algorithms hyperparameters.

D.1 Synthetic data generation

In this work, we rely on synthetic data to benchmark AdaScore's finite samples performance. For each dataset, we first sample the ground truth graph and then generate the observations according to the causal graph.

Erdös-Renyi graphs. The ground truth graphs are generated according to the Erdös-Renyi model. It allows specifying the number of nodes and the probability of connecting each pair of nodes). In ER graphs, a pair of nodes has the same probability of being connected.

Nonlinear causal mechanisms. Nonlinear causal mechanisms are parametrized by a neural network with random weights. We create a fully connected neural network with one hidden layer with 10 units, Parametric ReLU activation function, followed by one normalizing layer before the final fully connected layer. The weights of the neural network are sampled from a standard Gaussian distribution. This strategy for synthetic data generation is commonly adopted in the literature [26, 18, 28, 29, 27].

Linear causal mechanisms. For the linear mechanisms, we define a simple linear regression model predicting the effects from their causes and noise terms, weighted by randomly sampled coefficients. Coefficients are generated as samples from a Uniform distribution supported in the range [ -3 , -0 . 5] ∪ [0 . 5 , 3] . We avoid too small coefficients to avoid close to unfaithful datasets Uhler et al. [24].

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

Noise terms distribution. The noise terms are sampled from a Uniform distribution supported between -2 and 2 .

Finally, we remark that we standardize the data by their empirical data. This is known to remove shortcuts that allow finding a correct causal order sorting variables by their marginal variance, as in varsortability , described in Reisach et al. [32], or sorting variables by the magnitude of their score | ∂ X i log p ( X ) | , a phenomenon known as scoresortability analyzed by Montagna et al. [18].

D.2 AdaScore hyperparameters

For AdaScore, we set the α level for the required hypothesis testing at 0 . 05 . For the CAM-pruning step, the level is instead set at 0 . 001 , the default value of the dodidscover Python implementation of the method, and commonly found in all papers using CAM-pruning for edge selection [15, 16, 17, 36]. For the remaining parameters. The regression hyperparameters for the estimation of the residuals are found via cross-validation during inference: tuning is done minimizing the generalization error on the estimated residuals, without using the performance on the causal graph ground truth. Finally, for the score matching estimation, the regularization coefficients are set to 0 . 001 .

D.3 Computer resources

All experiments have been run on an AWS EC2 instance of type p3.2xlarge . These machines contain Intel Xeon E5-2686-v4 processors with 2.3 GHz and 8 virtual cores as well as 61 GB RAM. All experiments can be run within a day.

E Additional Experiments

In this section, we provide additional experimental results. All synthetic data has been generated as described in Appendix D.1.

E.1 Non-additive mechanisms

In Figure 1 we have demonstrated the performance of our proposed method on data generated by linear SCMs and non-linear SCMs with additive noise. But Proposition 1 also holds for any faithful distribution generated by an acyclic model. Thus, we employed as mechanism a neural network-based approach similar to the non-linear mechanism described in Appendix D. Instead of adding the noise term, we feed it as additional input into the neural network. Results in this setting are reported in Figure 2. As neither AdaScore nor any of the baseline algorithms has theoretical guarantees for the orientation of edges in this scenario, we report the F 1 -score (popular in classification problems) w.r.t. to the existence of an edge, regardless of orientation. Our experiments show that AdaScore can, in general, correctly recover the graph's skeleton in all the scenarios, with an F 1 score median between 1 and ∼ 0 . 75 , respectively for small and large numbers of nodes.

E.2 Sparse graphs

In this section, we present the experiments on sparse Erdös-Renyi graphs where each pair of nodes is connected by an edge with probability 0 . 3 . The results are illustrated in Figure 3. For sparse graphs, recovery results are similar to the dense case, with AdaScore generally providing comparable performance to the other methods.

E.3 Increasing number of samples

In the following series of plots we demonstrate the scaling behaviour of our method w.r.t. to the number of samples. Figure 5 shows results with edge probability 0.5 and Figure 4 with 0.3. All graphs contain seven observable nodes. As before we observe that AdaScore performs comparably to other methods. E.g. in Figures 4a and 5b we can see that the median error AdaScore improves with additional samples and in all plots we see that no other algorithm seems to gain an advantage over AdaScore with increasing sample size.

adascore

camuv

nogam

rcd

lingam

3

5

7

9

number of nodes

0.0

0.2

0.4

0.6

0.8

1.0

skeleton_f1

sparse

3

5

7

9

number of nodes

0.0

0.2

0.4

0.6

0.8

1.0

skeleton_f1

dense

(a) Fully observable model

3

5

7

9

number of nodes

0.0

0.2

0.4

0.6

0.8

1.0

skeleton_f1

sparse

3

5

7

9

number of nodes

0.0

0.2

0.4

0.6

0.8

1.0

skeleton_f1

dense

(b) Latent variables model

adascore

camuv

nogam

rcd

lingam

3

5

7

9

number of nodes

0

2

4

6

8

10

12

14

16

shd

linear

3

5

7

9

number of nodes

0

5

10

15

20

25

shd

nonlinear

(a) Fully observable model

3

5

7

9

number of nodes

0

5

10

15

20

25

shd

linear

3

5

7

9

number of nodes

0

5

10

15

20

25

shd

nonlinear

(b) Latent variables model

adascore

camuv

nogam

rcd

lingam

500

1000

1500

2000

number of samples

0

2

4

6

8

10

12

14

shd

linear

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

nonlinear

(a) Fully observable model

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

linear

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

nonlinear

(b) Latent variables model

adascore

camuv

nogam

rcd

lingam

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

nonlinear

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

shd

linear

(a) Fully observable model

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

nonlinear

500

1000

1500

2000

number of samples

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

shd

linear

(b) Latent variables model

E.4 Limitations 828

In this section, we remark the limitations of our empirical study. It is well known that causal discovery 829 lacks meaningful, multivariate benchmark datasets with known ground truth. For this reason, it is 830 common to rely on synthetically generated datasets. We believe that results on synthetic graphs should 831 be taken with care, as there is no strong reason to believe that they should mirror the benchmarked 832 algorithms' behaviors in real-world settings, where often there is no prior knowledge about the 833 structural causal model underlying available observations. 834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we claim that we connect the properties of the score function to causal structure learning. In the paper, particularly sections 3 and 4, we present the theoretical results supporting our claim. Further, in the abstract we mention that based on our theory we propose an algorithm for causal discovery from score matching estimation, algorithm that we define in Section 4.3 and we empirically validate in Section 5 and Appendix E.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main limitation of our work is on the experimental side: our experiments are limited to synthetic data, which are not an ideal probing ground. Additionally, our

method does not provide performance that clearly improves on the existing literature. These limitations of our work are discussed in the discussion of the experiments in Section 5, as well as in the "Limitations" appendix section E.4. Concerning the assumptions required by our method, we thoroughly discuss them in the theoretical sections of the paper, where we define the results that are later used for the definition of the AdaScore. Finally, computational complexity is discussed in Appendix C.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our theoretical results make explicit the assumptions for which they are valid. Plus, we in section Appendix B we provide the proofs of our theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Appendix D, we provide all the details to reproduce the data generation of our experiments and the hyperparameters used in AdaScore for our experimental runs.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code for the experiments and the data generation in a zip file. Further, we describe all the details for reproducing our experimental results in Appendix D.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In our experiments section 5, we present all the necessary details on the data generation procedure and a description of the empirical results that are necessary for understanding our findings. Additionally, a comprehensive overview of our experimental design is presented in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report all our experimental results in the form of boxplots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

Answer: [Yes]

Justification: Details on the computer resources required for the experiments can be found in Appendix D.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not believe any of the concerns in the Code of Ethics apply to our work. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: In this work, we present a novel causal discovery method from observational data. We believe there are no specific negative societal impacts, while positive impacts are those generally recognized to causal discovery, as discussed in the Introduction section 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. 1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We release a causal discovery model from observational data which does not poses such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use our proprietary assets, as well as public assets available under MIT license, which we correctly cite in our work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release the code for AdaScore, submitted in the form of a zip file containing the necessary documentation for usage. Moreover, an extensive description of the data generation is provided in the paper, as well as a description of the method itself.

Guidelines:

The answer NA means that the paper does not release new assets.

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not work with human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not work with human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present novel theoretical results to identify causal effects in restricted ANMs even in case of unobserved confounders.\n\nSTRENGTHS:\n**The paper provides novel contributions to the field of score-based causal discovery by extending previous works to confounded restricted ANMs. Based on these contributions, I strongly support this paper's acceptance.**\n\n- the problem definition is very useful to orient readers\n- the theoretical results are novel\n- the experiments compare against a sufficient number of baselines, and though the proposed method is not SOTA, it compares well and has better theoretical guarantees\n\n\nBased on the authors' response, I am eager to improve my score.\n\nWEAKNESSES:\nI have a few remarks on improving the flow of the paper; however, even the first four points are not considered major issues.\n\n- Even though condition 1 is a well-known result in the causality literature, I suggest explaining why that admits linear models and including some description of **restricted ANMs** in the main text (at least for me, it is not evident, especially since the condition lacks intuition). To be clear, even this point does not diminish the main contribution, which I see regarding the results for confounders.\n- As **inducing paths** are an important concept for the main contributions, please _include it in the main text_ if space permits (suggestion: you can reduce spacing in $\\texttt{itemize}$ by setting $\\texttt{\\\\\\\\begin\\\\{itemize\\\\}[nolistsep]}$ )\n- I could **not find the definition of an active path** (not even in Def. 5, where it is said to be defined); I presume it is a path that is not blocked, but it would be better to state this explicitly. Maybe it would even be better to use \"a path that is not blocked\" instead of introducing new terminology (this is the first time I encountered \"active paths\"; I could be wrong about this)\n- The text is sometimes difficult to follow, due to heavy reliance on notation. I'd consider delegating the not crucial part to the appendix (potential candidates in 2.1) and using the remaining space to explain the main quantities better, especially the residuals (e.g., Eq. 12)\n\n\n## Minor points\n- please specify what you are calculating the expectation with respect to (using a bold E is also unconventional, though it's clear from the context it's an expectation)\n- as the mathematical objects for d-/m-separation are distinguishable, you might consider dropping the superscript to simplify notation; also, I'd suggest adding whitespace after $\\perp^d_\\mathcal{G}$ and the like to make it easier for the reader to attribute the indices to $\\perp$ and not the not on its right\n- I could not find the definition for $\\dot{\\cup}$ \n- in the explanation of Prop 3., the wording makes it a bit hard to discern that you also provide intuition for the second part; it would help if you refer to _Part (ii)_ explicitly\n- _Score matching through the roof_ in the title does not have added value for me, I'd consider rephrasing it to convey the message that \"we propose score-based causal discovery methods for confounded restricted ANMs\"\n\nQUESTIONS:\nSome methods in the literature [1-4] use the Jacobian of a learned neural network instead of relying on the score for causal discovery/identifiability claims. How do you see the relation of your contribution to those Jacobian-based methods?\n\n\n- [1] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural DAG learning. In 8th International Conference on Learning Representations, ICLR 2020\n\n- [2] Lazar Atanackovic, Alexander Tong, Jason Hartford, Leo J. Lee, Bo Wang, and Yoshua Bengio. DynGFN: Bayesian Dynamic Causal Discovery using Generative Flow Networks, February 2023.\n\n- [3] Patrik Reizinger, Yash Sharma, Matthias Bethge, Bernhard Scholkopf, Ferenc Huszar, and Wieland Brendel. Jacobian-based Causal Discovery with Nonlinear ICA. Transactions on Machine Learning Research, April 2023.\n\n- [4] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A Linear NonGaussian Acyclic Model for Causal Discovery. Journal of Machine Learning Research, 7(10), 2006.\n\nLIMITATIONS:\nThe authors provide an **honest comparison** of their method in the experiments, clearly stating its limitations compared to other methods.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose AdaScore, a method for causal discovery that generalizes previous work based on score matching for SCMs with possibly latent nodes. They combine connections of the score to conditional independence as well as to additive noise SCMs and show that a NoGAM-type procedure works to recover the direction of non-confounded edges of the corresponding partial ancestral graph.\n\nSTRENGTHS:\n- Adapting NoGAM [1] to the case allowing hidden variables is a practically meaningful contribution.\n- Model assumptions are somewhat weakened compared to CAM-UV by allowing for general mechanisms within blocks of observed and latent parents.\n\nWEAKNESSES:\nThe novelty of the paper lies primarily in the application of NoGAM to orient very specific edges in a partial ancestral graph (PAG) (the ancestral graph that represents the Markov equivalence class, in analogue to a CPDAG). This falls significantly short of the main contributions as described by the authors on l33--55. Specifically:\n\n- The authors state that they show how constraints on the Jacobian of the score can be used as conditional independence testing. However, the extent to which this is done is only by noticing the equivalence between conditional independence and the corresponding zero in the Jacobian term (previously noted in [1,2]), without any formal analysis of the proposed t-test (Appendix C) as a statistical test of conditional independence (which happens to be a notoriously difficult test).\n\n- The authors state that their identification results for additive noise models generalize the previous results obtained by previous works. In l193, the authors state \"we remove the nonlinearity assumption (of [3]) and make the weaker hypothesis of a restricted additive noise model\", but 1) this is a stronger assumption than additive noise, not a weaker one, and 2) the authors in [3] also consider the same restricted additive noise model. \n\n- The authors claim that AdaScore is able to handle a broad class of causal models (l54), but three out of four possible situations are direct applications of existing work. 1) Under no structural assumptions with or without latent confounders, AdaScore simply performs constraint-based causal discovery (FCI) using the conditional independence properties of the Jacobian of the score, a straightforward application of [1] also previously noticed in [2]. 2) Under an additive noise assumption, AdaScore is exactly equivalent to NoGAM. 3) Only under an additive noise assumption with hidden confounders, does AdaScore generalize NoGAM to orient unconfounded edges of the PAG returned by FCI, which may be very few of the discovered adjacencies.\n\n### Other comments\n\n- The experiments do not seem to suggest that AdaScore out performs other methods in any meaningful way---in fact, in Figure 1 a) AdaScore is completely equivalent to NoGAM, and is thus redundant. In Figure 1 b), where AdaScore should distinguish itself, it does not appear to be consistently better than CAM-UV. \n\n- Much of the paper (> 5 pages) is spent on directly describing previous works, NoGAM[3] and/or provide basic background on DAGs and MAGs. \n\n[1] Spantini et al., \"Inference via low-dimensional couplings.\" JMLR 2018.\n[2] Montagna et al., \"Scalable causal discovery with score matching.\" CLeaR 2023.\n[3] Montagna et al., \"Causal discovery with score matching on additive models with arbitrary noise.\" CLeaR 2023.\n\nQUESTIONS:\n- Is it correct to say that the unconfounded edges that are oriented by AdaScore are the purely _undirected_ edges of the PAG, and not the potentially bidirected ones?\n\nLIMITATIONS:\nThe authors do not adequately discuss the limitations of their method---the limitations section in the appendix focuses purely on the empirical study. The authors claim that AdaScore is adaptive in the sense of being \"less reliant on prior assumptions which are often untestable\", but this is only in the sense that it performs different algorithms depending on user specification, which hardly constitutes one single unifying adaptive algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends theoretical results about causal discovery through score matching to encompass both linear and non-linear SCMs and lift the sufficiency assumption. The theoretical results relax the non-linearity assumption of Montagna et al 2023 by swapping it with the less restrictive one of restricted ANM from Peters et al 2009. As for the latent confounder detection a parallel with m-separation is drawn using results from Spantini et al 2018, to establish that the score will be non-zero in the presence of an active path. Following the theoretical results, an algorithm to estimate causal graphs from data is proposed and evaluated, generalizing the NoGAM algorithm of Montagna et al 2023, which only covers the non-linear case.\n\nSTRENGTHS:\nThe paper is clearly written and, if it wasn’t for some of the definitions relegated to the appendix, very easy to follow.  \n\nThe theoretical results, particularly Propositions 2 and 3 are important extensions of the score-matching methodology for causal discovery, dealing with both the non-linearity and sufficiency assumptions of the method proposed in Montagna et al. 2023.\n\nWEAKNESSES:\nThe paper motivation is basically the weakening of current assumptions for causal discovery methods. However, assumptions and benefits of the proposed methodology are not clearly specified. In line 74, the authors state that faithfulness is assumed (I believe, it is kind of hidden in the background notions). If that is the case, the method adopts the same assumption as FCI, plus ANM. So the proposed method is relaxing assumptions compared to CAM-UV, RCD and NoGAM, but adding onto FCI. Regarding the benefits, the alleged flexibility of the method to output DAGs, MECs, MAGs, PAGs, which should make it preferable to FCI, is merely touched upon in the contributions and the experiment section.  \n\nProposition 1 is a rather trivial application of the more general lemma in Spantini et al. and it does not specify the required faithfulness assumption to obtain the result from Eq. 6 in the paper. \n\nThe experimental results show limited added value according to the one metric chosen (SHD) in a synthetic setting. They are not comprehensive enough, with no application to common (pseudo-)real benchmarks (e.g. from bnlearn). More experiments and more metrics are needed as, as it stands, the proposed method seems to add no real value compared to the baselines. Additionally, it is not clear from the experiments if it is really able to identify confounders. Breaking down precision and recall by mark would show this. FCI and a random baseline should be also added for reference.  \n\nExperiments are conducted on data with at most 9 variables, and the scalability of the method is not shown nor discussed. \n\nThe model used to estimate residuals is not discussed, nor the assumption that the chosen model fits the data adequately to correctly estimate residuals, and what is needed to assess this.\n\nQUESTIONS:\n- Line 28: FCI weakness is highlighted to be that it outputs PAGs, does the proposed method improve on this at parity of assumptions? \n\n- Line 44: ANM is an assumption on the causal mechanism, is it not?\n\n- Line 217: \"the noise terms recentered by the latent causal effects\" what does this mean? \n\n- Line 221: \"N_i are assumed to be non-Gaussian when f_i is linear\" This reduces to RCD, right? \n\n- Line 257: \"the remaining arrowheads … are identified no better than in the equivalence class\" what does this mean? \n\n- Line 279: \"using hypothesis testing to find vanishing MSE...\" how is this set up? \n\n- Line 280: how much added value does pruning have? E.g. CAM pruning can have huge impact https://arxiv.org/pdf/1310.1533 \n\n- Alg. 1: do you start from a disconnected graph? Where do the \"remaining\" bidirected edges come from, do you reduce them prior to this? \n\n- Line 292: exogenous variables are selected at random, hence could be sink nodes with no consequence on the discovery process. Alternative and more meaningful testing strategies could be to select confounders (e.g. https://proceedings.neurips.cc/paper_files/paper/2021/file/144a3f71a03ab7c4f46f9656608efdb2-Paper.pdf) \n\n- Line 305: \"presents better performance\" I am not sure about this, they seem pretty aligned considering quartiles. Statistical tests to ascertain this are missing. \n\n- Line 315: some realistic scenarios are available through e.g. the bnlearn repository (https://www.bnlearn.com/bnrepository/). Any reason not to test on some of the data available there? \n\nMinor points and typos \n\n- Line 73: reference for d-separation is missing \n\n- Line 79: reference for MEC is missing \n\n- Line 130: have -> has \n\n- Line 147: \"directed global Markov property\": not introduced. I guess it is just the global Markov condition introduced before? \n\n- Footnote 1: provides -> provide \n\n- Line 162: \"unobserved active paths\" are not defined in Definition 5, m-separation.  \n\n- Line 209: full stop missing \n\n- Line 236: the PAG -> a PAG \n\n- Line 267: \"Appendix C.2\" what should the reader expect to find in there? \n\n- Line 296: \"We fix the […] level\" significance\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}