Title: An efficient search-and-score algorithm for ancestral graphs using multivariate information scores
Venue: NeurIPS 2024
Accepted: no
Keywords: causal discovery

search-and-score structure learning

latent variable

multivariate information
Abstract: We propose a greedy search-and-score algorithm for ancestral graphs, which include directed as well as bidirected edges, originating from unobserved latent variables. The normalized likelihood score of directed mixed graphs is estimated in terms of multivariate information over relevant subsets of variables, ${C}$, that are connected through collider paths confined to the ancestor set of ${C}$. For computational efficiency, the proposed two-step algorithm relies on local information scores limited to the close surrounding variables of each node (step 1)  and edge (step 2). This computational strategy is shown to outperform state-of-the-art causal discovery methods on challenging benchmark datasets.

1

Abstract

We propose a greedy search-and-score algorithm for ancestral graphs, which include directed as well as bidirected edges, originating from unobserved latent variables. The normalized likelihood score of ancestral graphs is estimated in terms of multivariate information over relevant subsets of vertices, C , that are connected through collider paths confined to the ancestor set of C . For computational efficiency, the proposed two-step algorithm relies on local information scores limited to the close surrounding vertices of each node (step 1) and edge (step 2). This computational strategy is shown to outperform state-of-the-art causal discovery methods on challenging benchmark datasets.

1 Introduction

The likelihood function plays a central role in the selection of a graphical model G based on 11 observational data D . Given N independent samples from D , the likelihood L D|G that they might 12 have been generated by the graphical model G is given by [1], 13

where H ( p, q ) = -∑ x p ( x ) log q ( x ) is the cross-entropy between the empirical probability distribu14 tion p ( x ) of the observed data D and the theoretical probability distribution q ( x ) of the model G and 15 Z D , G a data- and model-dependent factor ensuring proper normalization condition for finite dataset. In 16 short, Eq.1 results from the asymptotic probability that the N independent samples, x (1) , · · · , x ( N ) , 17 are drawn from the model distribution, q ( x ) , i.e. L D|G ≡ q ( x (1) , · · · , x ( N ) ) = ∏ i q ( x ( i ) ) , rather 18 than the empirical distribution, p ( x ) . This leads to, log L D|G = ∑ i log q ( x ( i ) ) , which converges 19 towards N ∑ x p ( x ) log q ( x ) = -N H ( p, q ) in the large sample size limit, N → ∞ , with 20 log Z D , G = O (log N ) . 21 The structural constraints of the model G translate into the factorization form of the theoretical 22 probability distribution, q ( x ) [2-6]. In particular, the probability distribution of Bayesian networks 23 (BN) factorizes in terms of conditional probabilities of each variable given its parents, as q BN ( x ) = 24 ∏ i q ( x i | pa X i ) , where pa X i denote the values of the parents of node X i in G , Pa X i . For Bayesian 25 networks, the factors of the model distribution, q ( x i | pa X i ) , can be directly estimated with the 26 empirical conditional probabilities of each node given its parents as, q ( x i | pa X i ) ≡ p ( x i | pa X i ) , 27 leading to the well known estimation of the likelihood function in terms of conditional entropies 28 H ( X i | Pa X i ) = -∑ x p ( x i , pa X i ) log p ( x i | pa X i ) , 29

This paper concerns the experimental setting for which some variables of the underlying Bayesian 30 model are not observed. This frequently occurs in practice for many applications. We derive an 31 explicit likelihood function for the class of ancestral graphs, which include directed as well as 32 bidirected edges, arising from the presence of unobserved latent variables. Tian and Pearl 2002 [7] 33 showed that the probability distribution of such graphs factorizes into c-components including subsets 34 of variables connected through bidirected paths ( i.e. containing only bidirected edges). Richardson 35 2009 [6] later proposed a refined factorization of the model distribution of the broader class of acyclic 36 directed mixed graphs in terms of conditional probabilities over 'head' and 'tail' subsets of variables 37 within each ancestrally closed subsets of vertices. However, unlike with Bayesian networks, the 38 contributions of c-components or head-and-tail factors to the likelihood function cannot simply be 39 estimated in terms of empirical distribution p ( x ) , as shown below. This leaves the likelihood function 40 of ancestral graphs difficult to estimate from empirical data, in general, although iterative methods 41 have been developped when the data is normally distributed [8-13]. 42

43

44

45

46

47

48

49

50

51

52

53

The present paper provides an explicit decomposition of the likelihood function of ancestral graphs in terms of multivariate cross-information over relevant ' ac -connected' subsets of variables, Figs. 1., which do not rely on the head-and-tail factorization but coincide with the parametrizing sets [14] derived from the head-and-tail factorization. It suggests a natural estimation of these revelant contributions to the likelihood function in terms of empirical distribution p ( x ) . This result extends the likelihood expression of Bayesian Networks (Eq. 2) to include the effect of unobserved latent variables and enables the implementation of a greedy search-and-score algorithm for ancestral graphs. For computational efficiency, the proposed two-step algorithm relies on local information scores limited to the close surrounding vertices of each node (step 1) and edge (step 2). This computational strategy is shown to outperform state-of-the-art causal discovery methods on challenging benchmark datasets.

54

55

56

57

58

59

60

61

62

63

64

65

2 Theoretical results

2.1 Multivariate cross-entropy and cross-information

The theoretical result of the paper (Theorem 1) is expressed in terms of multivariate cross-information derived from multivariate cross-entropies through the Inclusion-Exclusion Principle. The same expressions can be written between multivariate information and multivariate entropies by simply substituting q ( { x i } ) with p ( { x i } ) in the equations below and will be used to estimate the likelihood function of ancestral graphs (Proposition 3).

As recalled above, the cross-entropy between m variables, V = { X 1 , · · · , X m } , is defined as,

where p ( { x i } ) is the empirical joint probability distribution of the variables { X i } and q ( { x i } ) the joint probability distribution of the model. Bayes formula, q ( { x i } , { y j } ) = q ( { x i }|{ y j } ) q ( { y j } ) , directly translates into the definition of conditional cross-entropy through the decomposition,

Multivariate (cross) information, I ( V ) ≡ I ( X 1 ; · · · ; X m ) , are defined from multivariate (cross) 66 entropies through Inclusion-Exclusion formulas over all subsets of variables [15-18] as, 67

where the semicolon separators are needed to distinguish multipoint (cross) information from joint 68 variables as { X,Z } in I ( { X,Z } ; Y ) = I ( X ; Y )+ I ( Z ; Y ) -I ( X ; Y ; Z ) . Below, implicit separators 69 between non-conditioning variables in multivariate (cross) information will always correspond to 70 semicolons, e.g. as in I ( V ) in Eq. 5. Unlike multivariate (cross) entropies, which are always positive, 71

H ( X 1 , · · · , X k ) glyph[greaterorequalslant] 0 , multivariate (cross) information, I ( X 1 ; · · · ; X k ) , can be positive or negative 72 for k glyph[greaterorequalslant] 3 , while they remain always positive for k < 3 , i.e. I ( X ; Y ) glyph[greaterorequalslant] 0 and I ( X ) glyph[greaterorequalslant] 0 . 73

In turn, multivariate (cross) entropies can be expressed through the Principle of Inclusion-Exclusion 74 into the same expression form but in terms of multivariate (cross) information, 75

Conditional multivariate (cross) information I ( V | Z ) are defined similarly as multivariate (cross) 76 information I ( V ) but in terms of conditional (cross) entropies as, 77

Eqs. 5 & 7 lead to a decomposition rule relative to a variable Z , Eq. 8, which can be conditioned 78 on a set of joint variables, A = { A 1 , · · · , A m } , with implicit comma separators for conditioning 79 variables in Eq. 9, 80

Alternatively, conditional (cross) information, such as I ( X ; Y | A ) , can be expressed in terms of 81 non-conditional (cross) entropies using Eq. 4, 82

which can in turn be expressed in terms of non-conditional (cross) information as, 83

where S = { X,Y } ∪ A . This corresponds, up to an opposite sign, to all (cross) information terms 84 including both X and Y in the expression of the multivariate (cross) entropy, H ( X,Y, A ) , Eq. 6. 85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

2.2 Graphs and connection criteria

2.2.1 Directed mixed graphs and ancestral graphs

Two vertices are said to be adjacent if there is an edge (of any type) between them, X ∗ ∗ Y , where ∗ stands for any (head or tail) end mark. X and Y are said to be neighbors if X Y , parent and child if X → Y and spouses if X ←→ Y in G .

A path in G is a sequence of distinct vertices V 1 , . . . , V n consecutively adjacent in G , as, V 1 ∗ ∗ V 2 ∗ ∗ · · · ∗ ∗ V n -1 ∗ ∗ V n . In particular, a collider path between V 1 and V n has the form V 1 ∗→ V 2 ←→··· ←→ V n -1 ←∗ V n and a directed path corresponds to V 1 → V 2 →··· → V n .

X is called an ancestor of Y and Y a descendant of X if X = Y or there is a directed path from X to Y , X → ··· → Y . An G ( Y ) denotes the set of ancestors of Y in G . By extension, for any subset of vertices, C ⊆ V , An G ( C ) denotes the set of ancestors for all Y ∈ C in G .

A directed mixed graph is a vertex-edge graph G = ( V , E ) that can contain two types of edges: directed ( → ) and bidirected ( ←→ ) edges.

A directed cycle occurs in G when X ∈ An G ( Y ) and X ← Y . An almost directed cycle occurs when X ∈ An G ( Y ) and X ←→ Y .

Definition 1. An ancestral graph is a directed mixed graph:

without directed cycles;

ii) without almost directed cycles.

An ancestral graph is said to be maximal if every missing edge corresponds to a structural independence. If an ancestral graph G is not maximal, there exists a unique maximal ancestral graph ¯ G by adding bidirected edges to G [8].

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

2.2.2 ac -connecting paths and ac -connected subsets

Let us now define ancestor collider connecting paths or ac -connecting paths , which entail simpler path connecting criterion than the traditional m-connecting criterion , discussed in the Appendix A. Yet, ac -connecting paths and ac -connected subsets will turn out to be directly relevant to characterize the likelihood decomposition and Markov equivalent classes of ancestral graphs.

Definition 2 . [ ac -connecting path] An ac -connecting path between X and Y given a subset of variables C (possibly including X and Y ) is a collider path, X ∗→ Z 1 ←→ ··· ←→ Z K ←∗ Y , with all Z i ∈ An G ( { X,Y } ∪ C ) , that is, with Z i in C or connected to { X,Y } ∪ C by an ancestor path, i.e. Z i →··· → T with T ∈ { X,Y } ∪ C .

Definition 3 . [ ac -connected subset] A subset C is said to be ac -connected if ∀ X,Y ∈ C , X and Y are connected (through any type of edge) or there is an ac -connecting path between X and Y given C .

2.3 Likelihood decomposition of ancestral graphs

Theorem 1. [ likelihood of ancestral graphs ] The cross-entropy H ( p, q ) and likelihood L D|G of an ancestral graph G is decomposable in terms of multivariate cross-information, I ( C ) , summed over all ac -connected subsets of variables, C (Definition 3),

where N is the number of iid samples in the dataset D and Z D , G a data- and model-dependent normalization constant.

The proof of Theorem 1 is left to the Appendix B. It is based on a partition of the cross-entropy (Eq. 6) into cross-information contributions from ac -connected and nonac -connected subsets of variables, which do not rely on head-and-tail factorizations. Hu and Evans [14] proposed an equivalent result (Proposition 3.3 in [14]) with a proof using head-and-tail decomposition to define parametrizing sets, which happen to coincide with the ac -connected sets defined here (Definition 3). Theorem 1 characterizes in particular the Markov equivalence class of ancestral graphs [8, 19-24] as,

Corollary 2. Two ancestral graphs are Markov equivalent if and only if they have the same ac -connected subsets of vertices.

Note, in particular, that Eq. 12 holds for maximal ancestral graphs (MAG), for which all pairs of ac -connected variables are connected by an edge, and their Markov equivalent representatives, the partial ancestral graphs (PAG) [8, 25-27].

Proposition 3. The likelihood decomposition of ancestral graphs (Eq. 12, Theorem 1) can be estimated by replacing the model distribution q by the empirical distribution p in the retained multivariate cross-information terms I ( C ) corresponding to all ac -connected subsets of variables, C .

Hence, Proposition 3 amounts to estimating all relevant cross-information terms in the likelihood function with the corresponding multivariate information terms computed from the available data, while assuming by construction that the model distribution obeys all local and global conditional independences entailed by the ancestral graph. The corresponding factorization of the model distribution can be expressed in terms of empirical distribution, assuming positive distributions, see Appendix C.

Fig. 1 illustrates the cross-entropy decomposition for a few graphical models in terms of crossinformation contributions from their ac -connected subsets of vertices. In particular, an unshielded non-collider ( e.g. X → Z → W , Fig. 1A), is less likely ( i.e. higher cross-entropy) than an unshielded collider or 'v-structure' ( e.g. X → Z ← W , Fig. 1B), if the corresponding three-point information term is negative, I ( X ; Z ; W ) < 0 , in agreement with earlier results [28, 29]. However, this early approach, exploiting the sign and magnitude of three-point information to orient v-structures, does not include higher order terms involving multiple v-structures, which can lead to orientation conflicts between unshielded triples, in practice. Resolving such orientation conflicts requires to include

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

Z

Y

T

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

Z

Y

T

X

Z

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

Y

T

X

Y

T

Z

Y

T

X

Z

X

T

+

X

Z

Y

T

-

-

-

Z

Y

T

X

Z

T

X

Z

T

Z

Y

T

Z

Y

T

Z

Y

T

X

i

X

j

X

k

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

X

i

X

j

X

k

X

l

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

B

C

A

X

i

X

i

X

j

- Σ

Edges

I(   ;    )

=

Σ

I(    )

Vertices

G

H

I

F

E

-

-

W

Z

X

W

Z

X

T

Y

Z

X

T

Y

Z

X

T

Y

Z

X

T

Y

Z

X

+

+

+

T

X

+ Σ

I(   ;    ;    )

H(p,q)

ac-connected

Triples

+

X

Z

W

Y

X

Z

W

Y

X

Z

W

Y

I(   ;    ;    ;    )

4-uples

ac-connected

D

X

Z

Y

W

+

+

X

Z

Y

W

-

W

X

Z

Y

Y

Z

W

+

Z

Y

W

X

Z

Y

X

Y

Z

W

X

Z

Y

Z

Y

W

W

X

Y

X

Z

W

+ ...

Z

Y

W

- Σ

X

X

Z

Y

information contributions from higher-order ac -connected subgraphs, such as star-like ac -connected 152 subsets including three or more parents, Fig. 1C. Similarly, the cross-entropies of collider paths 153 involving several colliders also include higher-order terms, as with the simple example of a two154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

collider path, Fig. 1D. By contrast, the cross-entropy based on the head-and-tail factorization of the same two-collider path, i.e. q ( x, z, y, w ) = q ( z, y | x, w ) q ( x ) q ( w ) [6], is found to be equivalent to the cross-entropy of a Bayesian graph without bidirected edge, Fig. 1E, when estimated with the empirical distribution p ( . ) , see Appendix C. This observation illustrates the difficulty to estimate the likelihood functions of ancestral graphs using head-and-tail factorization.

Further examples of graphical models, Figs. 1F-I, show the relative simplicity of the decomposition with only few (non-trivial) ac -connected contributing subsets C with | C | glyph[greaterorequalslant] 3 , as compared to the much larger number of nonac -connected non-contributing subsets, that cancel each other by construction due to conditional independence constraints of the underlying model. Note, in particular, that most contributing multivariate information I ( C ) only concern direct connections or collider paths within a single component subgraph induced by C (solid line edges in Fig. 1). However, occasionally, collider paths extending beyond C into An G ( C ) \ C (marked with wiggly edges) with corresponding ancestor path(s) (marked with dashed edges) do occur, as shown in Fig. 1G.

In addition, the present information-theoretic decomposition of the likelihood of ancestral graphs can readily distinguish their Markov equivalence classes according to Corollary 2. For instance, the ancestral graphs of Fig. 1F and Fig. 1G, despite sharing the same edges and the same unshielded collider ( X → Z ← T ), turn out not to be Markov equivalent, as discussed in [24]. Indeed, their cross-entropy decompositions differ by two ac -connected contributing terms: a three-point cross information I ( X ; Y ; T ) with a collider path not confined in C ( i.e. X glyph[squiggleright] Z glyph[squiggleleftright] T ←→ Y and corresponding ancestor path Z glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Y ) and a four-point information term I ( X ; Y ; Z ; T ) due to the two-collider path ( X → Z ←→ T ←→ Y ). More quantitatively, it shows that the graph of Fig. 1G with a two-collider path is more likely than the graph of Fig. 1F whenever I ( X ; Y ; T ) -I ( X ; Y ; Z ; T ) = I ( X ; Y ; T | Z ) = I ( X ; Y | Z ) -I ( X ; Y | Z, T ) < 0 . Finally, the Markov equivalent graphs of Fig. 1H and Fig. 1I, also due to [24], illustrate the fact that the actual ancestor collider path between unconnected pairs does not need to be unique nor conserved between Markov equivalent graphs (as long as their cross-entropies share the same multivariate cross-information decomposition).

3 Efficient search-and-score causal discovery using local information scores

The likelihood estimation of ancestral graphs (Theorem 1 and Proposition 3) enables the implementation of a search-and-score algorithm for this broad class of graphs, which has attracted a number of contributions recently [11-13, 30-32]. Our specific objective is not to develop an exact method limited to simple graphical models with a few nodes and small datasets but to implement an efficient and reliable heuristic method applicable to more challenging graphical models and large datasets.

Indeed, search-and-score structure learning methods need to rely on heuristic rather than exhaustive search, in general, given that the number of ancestral graphs grows super-exponentially as the number of vertices increases. This can be implemented for instance with a Monte Carlo algorithmic scheme with random restarts, which efficiently probes relevant graphical models. Here, we opt, instead, to use the prediction of an efficient hybrid causal discovery method, MIIC [29, 33, 34], as starting point for a subsequent search-and-score approach based on the proposed likelihood estimation of ancestral graphs (Eq. 12 and Proposition 3).

Moreover, while the likelihood decomposition of ancestral graphs may involve extended ac -connected subsets of variables, as illustrated in Fig. 1, we aim to implement a computationally efficient searchand-score causal discovery method based on approximate local scores limited to the close surrounding vertices of each node and edge. Yet, while MIIC only relies on unshielded triple scores, the novel search-and-score extension, MIIC_search&score, uses also higher-order local information scores to compare alternative subgraphs, as detailed below.

The proposed method is shown to outperform MIIC and other state-of-the-art causal discovery methods on challenging datasets including latent variables.

3.1 MIIC, an hybrid causal discovery method based on unshielded triple scores

MIIC is an hybrid causal discovery method combining constraint-based and information-theoretic frameworks [29, 35]. Unlike traditional constraint-based methods [4, 5], MIIC does not directly attempt to uncover conditional independences but, instead, iteratively substracts the most significant three-point (conditional) information contributions of successive contributors, A 1 , A 2 , ..., A n , from

the mutual information between each pair of variables, I ( X ; Y ) , as, 207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

where I ( X ; Y ; A k |{ A i } k -1 ) > 0 is the positive information contribution from A k to I ( X ; Y ) [28, 36]. Conditional independence is eventually established when the residual conditional mutual information on the right hand side of Eq. 13, I ( X ; Y |{ A i } n ) , becomes smaller than a complexity term, i.e. k X ; Y |{ A i } ( N ) glyph[greaterorequalslant] I ( X ; Y |{ A i } n ) glyph[greaterorequalslant] 0 , which dependents on the considered variables and sample size N .

This leads to an undirected skeleton, which MIIC then (partially) orients based on the sign and amplitude of the regularized conditional 3-point information terms [28, 29]. In particular, negative conditional 3-point information terms, I ( X ; Y ; Z |{ A i } ) < 0 , correspond to the signature of causality in observational data [28] and lead to the prediction of a v-structure, X → Z ← Y , if X and Y are not connected in the skeleton. By contrast, a positive conditional 3-point information term, I ( X ; Y ; Z |{ A i } ) > 0 , implies the absence of a v-structure and suggests to propagate the orientation of a previously directed edge X → Z Y as X → Z → Y .

In practice, MIIC's strategy to circumvent spurious conditional independences significantly improves recall, that is, the fraction of correctly recovered edges, compared to traditional constraint-based methods [28, 29]. Yet, MIIC only relies on unshielded triple scores to reliably uncover significant contributors and orient v-structures, as outlined above. MIIC has been recently improved to ensure the consistency of the separating set in terms of indirect paths in the final skeleton or (partially) oriented graphs [37, 34] and to improve the reliably of predicted orientations [33, 34].

The predictions of this recent version of MIIC, which include three type of edges (directed, bidirected and undirected), have been used as starting point for the subsequent local search-and-score method implemented in the present paper.

3.2 New search-and-score method based on higher-order local information scores

Starting from the structure predicted by MIIC, as detailed above, MIIC_search&score method proceeds in two steps.

3.2.1 Step 1: Node scores for edge orientation priming and edge removal

The first step consists in minimizing a node score corresponding to the local normalized log likelihood of each node w.r.t. its possible parents or spouses amongst the connected nodes predicted by MIIC. To this end, the node score assesses the conditional entropy of each node w.r.t. a selection of parents, spouses or neighbors, Pa ′ Xi ⊆ Pa Xi ∪ Sp Xi ∪ Ne Xi , and a factorized Normalized Maximum Likelihood (fNML) regularization [28], see Appendix D for details,

where q x i corresponds to the combination of levels of Pa ′ Xi , while r x i is the number of levels of X i , and n j the number of samples corresponding to a particular combination of levels j in each summand, with ∑ j n j = N , the total number of samples. log C r x i n j is the fNML regulatization cost summed over all combinations of levels, q x i , [38, 39], see Appendix D.

This first algorithm is looped over each node, priming the orientations of their surrounding edges (as directed, bidirected or undirected), until convergence. Edges without orientation priming at either extremity are removed at the end of Step 1.

3.2.2 Step 2: Edge orientation scores

The second step consists in minimizing an edge orientation score corresponding to the local normalized log likelihood of each edge w.r.t. its nodes' parents and spouses inferred in Step 1. To this end, the edge score assesses the conditional information and a fNML complexity cost with respect to the type of orientation, given three sets of parents and spouses of X and Y , i.e. Pa ′ X \ Y = Pa X ∪ Sp X \ Y , Pa ′ Y \ X = Pa Y ∪ Sp Y \ X and Pa ′ XY = Pa ′ X \ Y ∪ Pa ′ Y \ X with their corresponding combinations of

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

levels, q y \ x , q x \ y and q xy . These orientation scores, listed in Table 1, include symmetrized fNML complexity terms to enforce Markov equivalence, if X and Y share the same parents or spouses (excluding X and Y ), see Appendix D. Indeed, all three scores become equals if Pa ′ Y \ X = Pa ′ X \ Y = Pa ′ XY implying also the same combinations of parent and spouse levels, q y \ x = q x \ y = q xy .

This second algorithm is looped over each edge to compute an orientation score decrement, given the orientations of its surrounding edges. The orientation change corresponding to the largest orientation score decrement is then chosen at each iteration until convergence or until a limit cycle is reached and stopped at the lowest sum of local orientation scores.

4 Experimental results

We first tested whether MIIC_search&score orientation scores (Table 1) effectively predicts bidirected orientations on three simple ancestral models, Fig. 3, when the end nodes do not share the same parents (Fig. 3, Model 1), share some parents (Fig. 3, Model 2) or when the bidirected edge is part of a longer than two-collider paths (Fig. 3, Model 3). The prediction of the edge orientation scores are summarized in Table 3, Appendix E, and show good predictions for large enough datasets.

Beyond these simple examples, focussing on the discovery of bidirected edges in small toy models of ancestral graphs, we also analyzed more challenging benchmarks from the bnlearn repository [40], Fig. 2. They concern ancestral graphs obtained by hiding up to 20% of variables in Bayesian Networks of increasing complexity (number of nodes and parameters), such as Alarm (37 nodes, 46 links, 509 parameters), Insurance (27 nodes, 52 links, 984 parameters), and Barley (48 nodes, 84 links, 114,005 parameters). We then assessed causal discovery performance in terms of Precision , TP/ ( TP + FP ) , and Recall , TP/ ( TP + FN ) , relative to the theoretical PAGs, while counting as false positive ( FP ), all correctly predicted edges but without or with a different orientation as the directed or bidirected edges of the PAG.

Fig. 2 compares MIIC_search&score performance to MIIC results used as starting point for MIIC_search&score and to FCI [41]. MIIC and MIIC_search&score settings were set as described in section 3 above. The open-source MIIC R package (v1.5.2, GPL-3.0 license) was obtained at https://github.com/miicTeam/miic_R_package . FCI from the python causal-learn package (v0.1.3.8, MIT license) [41] was obtained at https://github.com/py-why/causal-learn and run with G 2 -conditional independence test and default parameter α = 0 . 05 .

Overall, MIIC_search&score is found to outperform MIIC in terms of edge precision with little to no decrease in edge recall, Fig. 2, demonstrating the benefit of MIIC_search&score's rationale to improve MIIC predictions by extending MIIC information scores from unshielded triples to higher-order information contributions. These originate from ac -connected subsets including nodes with more than two parents or spouses, or ac -connected subsets including two-collider paths. MIIC_search&score is also found to outperform FCI on complex ancestral benchmark networks with many parameters, such as Barley (114,005 parameters), Fig. 2. However, FCI is found to reach similar or better precision scores on easier benchmarks with fewer parameters ( i.e. Alarm and Insurance), although its recall remains usually lower than MIIC_search&score, especially at small sample size, as expected for a purely constraint-based causal discovery approach.

Importantly, the benchmark PAGs used to score the causal discovery results with increasing proportions of latent variables, Fig. 2, include not only bidirected edges originating from hidden common causes but also additional directed or undirected edges arising, in particular, from indirect effects of

Precision

Recall

Precision

Recall

Precision

Recall

1,000

10,000

100

1,000

10,000

100

1,000

10,000

100

1,000

10,000

1,000

10,000

100

1,000

10,000

0

1

0

1

0

1

ALARM  (509 parameters)

BARLEY  (114,005 parameters)

100

Sample  Size  (N)

100

0

1

20 %  LV

10 %  LV

5 %  LV

0 %  LV

INSURANCE  (1,008 parameters)

H

1H

fci

mic_search_

and_

score

hidden variables with observed parents. Irrespective of their orientations, all these additional edges 293 originating from indirect effects of hidden variables generally correspond to weaker effects ( i.e. lower 294 mutual information of indirect effects due to the Data Processing Inequality) and are more difficult to 295 uncover than the edges of the original graphical model without hidden variables. 296

297

298

299

5 Limitations

The main limitation of the paper concerns the local scores used in the search-and-score algorithm, which are limited to ac -connected subsets of vertices with a maximum of two-collider paths.

While this approach could be extended to higher-order information contributions including three300 or more collider paths, it allows for a simple two-step search-and-score scheme at the level of 301 individual nodes (step 1) and edges (step 2), as detailed in section 3. This already shows a significant 302 improvement in causal discovery performance ( i.e. combing good precision and good recall on 303 challenging benchmarks) as compared to existing state-of-the-art methods. 304

305

306

307

308

References

D. Koller, N. Friedman, Probabilistic Graphical Models: Principles and Techniques (MIT Press, 2009).

J. Pearl, A. Paz, Graphoids: A graph-based logic for reasoning about relevance relations, or when would x tell you more about y if you already know z, Tech. rep. , UCLA Computer Science Department (1985).

J. Pearl, Probabilistic reasoning in intelligent systems (Morgan Kaufmann, San Mateo, CA, 1988). 309

J. Pearl, Causality: models, reasoning and inference (Cambridge University Press, 2009), second edn. 310

P. Spirtes, C. Glymour, R. Scheines, Causation, Prediction, and Search (MIT press, , 2000), second edn. 311

T. S. Richardson, A factorization criterion for acyclic directed mixed graphs, Proceedings of the Twenty312 Fifth Conference on Uncertainty in Artificial Intelligence , UAI '09 (AUAI Press, Arlington, VA, USA, 313 2009), p. 462-470. 314

J. Tian, J. Pearl, A general identification condition for causal effects, Proceedings of the National Confer315 ence on Artificial Intelligence (Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 316 2002), pp. 567-573. 317

T. Richardson, P. Spirtes, Ancestral graph markov models. Ann. Statist. 30 , 962-1030 (2002). 318

M. Drton, M. Eichler, T. S. Richardson, Computing maximum likelihood estimates in recursive linear 319 models with correlated errors. Journal of Machine Learning Research 10 , 2329-2348 (2009). 320

R. J. Evans, T. S. Richardson, Maximum likelihood fitting of acyclic directed mixed graphs to binary 321 data, Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence , UAI'10 (AUAI Press, 322 Corvallis, OR, USA, 2010). 323

S. Triantafillou, I. Tsamardinos, Score-based vs constraint-based causal learning in the presence of 324 confounders, CFA@UAI (2016). 325

K. Rantanen, A. Hyttinen, M. Järvisalo, Maximal ancestral graph structure learning via exact search, 326 Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , C. de Campos, 327 M. H. Maathuis, eds. (PMLR, 2021), vol. 161 of Proceedings of Machine Learning Research , pp. 1237328 1247. 329

T. Claassen, I. G. Bucur, Greedy equivalence search in the presence of latent confounders, Proceedings of 330 the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , J. Cussens, K. Zhang, eds. (PMLR, 331 2022), vol. 180 of Proceedings of Machine Learning Research , pp. 443-452. 332

Z. Hu, R. Evans, Faster algorithms for markov equivalence, Proceedings of the 36th Conference on Uncer333 tainty in Artificial Intelligence (UAI) , J. Peters, D. Sontag, eds. (PMLR, 2020), vol. 124 of Proceedings of 334 Machine Learning Research , pp. 739-748. 335

W. J. McGill, Multivariate information transmission. Trans. of the IRE Professional Group on Information 336 Theory (TIT) 4 , 93-111 (1954). 337

H. K. Ting, On the amount of information. Theory Probab. Appl. 7 , 439-447 (1962). 338

T. S. Han, Multiple mutual informations and multiple interactions in frequency data. Information and 339 Control 46 , 26-45 (1980). 340

341

342

R. W. Yeung, A new outlook on shannon's information measures. IEEE transactions on information theory 37 , 466-474 (1991).

P. Spirtes, T. Richardson, A polynomial time algorithm for determinint dag equivalence in the presence of 343 latent variables and selection bias, Proceedings of the 6th International Workshop on Artificial Intelligence 344 and Statistics (1996). 345

346

347

T. Richardson, Markov properties for acyclic directed mixed graphs. Scandinavian Journal of Statistics 30 , 145-157 (2003).

348

349

350

R. A. Ali, T. S. Richardson, Markov equivalence classes for maximal ancestral graphs, Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence , UAI'02 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2002), pp. 1-9.

R. A. Ali, T. S. Richardson, P. Spirtes, J. Zhang, Towards characterizing markov equivalence classes for 351 directed acyclic graphs with latent variables, Proceedings of the Fifteenth Conference on Uncertainty in 352 Artificial Intelligence , UAI'05 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2005). 353

J. Tian, Generating markov equivalent maximal ancestral graphs by single edge replacement, Proceedings 354 of the Fifteenth Conference on Uncertainty in Artificial Intelligence , UAI'05 (Morgan Kaufmann Publishers 355 Inc., San Francisco, CA, USA, 2005). 356

357

358

R. A. Ali, T. S. Richardson, P. Spirtes, Markov equivalence for ancestral graphs. Ann. Statist. 37 , 2808-2837 (2009).

359

360

361

362

363

364

365

366

367

368

369

370

371

372

T. Richardson, P. Spirtes, Scoring ancestral graph models, Tech. rep. (1999). Available as Technical Report CMU-PHIL 98.

J. Zhang, A characterization of markov equivalence classes for directed acyclic graphs with latent variables, Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence , UAI'07 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2007).

J. Zhang, On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artif. Intell. 172 , 1873-1896 (2008).

S. Affeldt, H. Isambert, Robust reconstruction of causal graphical models based on conditional 2-point and 3-point information, Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI 2015, July 12-16, 2015, Amsterdam, The Netherlands (2015), pp. 42-51.

L. Verny, N. Sella, S. Affeldt, P. P. Singh, H. Isambert, Learning causal networks with latent variables from multivariate information in genomic data. PLoS Comput. Biol. 13 , e1005662 (2017).

B. Andrews, G. F. Cooper, T. S. Richardson, P. Spirtes, The m-connecting imset and factorization for admg models, Preprint (2022). Arxiv 2207.08963.

Z. Hu, R. J. Evans, Towards standard imsets for maximal ancestral graphs. Bernoulli 30 (2024). 373

Z. Hu, R. Evans, A fast score-based search algorithm for maximal ancestral graphs using entropy, Preprint 374 (2024). Arxiv 2402.04777. 375

376

377

378

V. Cabeli, H. Li, M. da Câmara Ribeiro-Dantas, F. Simon, H. Isambert, Reliable causal discovery based on mutual information supremum principle for finite datasets, WHY21, 35rd Conference on Neural Information Processing Systems (NeurIPS, 2021).

379

380

381

382

383

384

385

M. d. C. Ribeiro-Dantas, H. Li, V. Cabeli, L. Dupuis, F. Simon, L. Hettal, A.-S. Hamy, H. Isambert, Learning interpretable causal networks from very large datasets, application to 400, 000 medical records of breast cancer patients. iScience 27 , 109736 (2024).

V. Cabeli, L. Verny, N. Sella, G. Uguzzoni, M. Verny, H. Isambert, Learning clinical networks from medical records based on information estimates in mixed-type data. PLoS Comput. Biol. 16 , e1007866 (2020).

S. Affeldt, L. Verny, H. Isambert, 3off2: A network reconstruction algorithm based on 2-point and 3-point information statistics. BMC Bioinformatics 17 (2016).

H. Li, V. Cabeli, N. Sella, H. Isambert, Constraint-based causal structure learning with consistent separating 386 sets. Advances in Neural Information Processing Systems (NeurIPS) 32 (2019). 387

P. Kontkanen, P. Myllymäki, A linear-time algorithm for computing the multinomial stochastic complexity. 388 Inf. Process. Lett. 103 , 227-233 (2007). 389

T. Roos, T. Silander, P. Kontkanen, P. Myllymäki, Bayesian network structure learning using factorized 390 nml universal models, Proc. 2008 Information Theory and Applications Workshop (ITA-2008) (IEEE Press, 391 2008). 392

M. Scutari, Learning Bayesian Networks with the bnlearn R Package. J. Stat. Softw. 35 , 1-22 (2010). 393

Y. Zheng, B. Huang, W. Chen, J. Ramsey, M. Gong, R. Cai, S. Shimizu, P. Spirtes, K. Zhang, Causal-learn: 394 Causal discovery in python. Journal of Machine Learning Research 25 , 1-8 (2024). 395

Y. M. Shtarkov, Universal sequential coding of single messages. Problems of Information Transmission 23 , 396 3-17 (1987). 397

J. Rissanen, I. Tabus, Adv. Min. Descrip. Length Theory Appl. (MIT Press, 2005), pp. 245-264. 398

W. Szpankowski, Average case analysis of algorithms on sequences (John Wiley & Sons, , 2001). 399

P. Kontkanen, W. Buntine, P. Myllymäki, J. Rissanen, H. Tirri, Efficient computation of stochastic 400 complexity. in: C. Bishop, B. Frey (Eds.) Proceedings of the Ninth International Conference on Artificial 401 Intelligence and Statistics, Society for Artificial Intelligence and Statistics 103 , 233-238 (2003). 402

P. Kontkanen, Computationally efficient methods for mdl-optimal density estimation and data clustering, 403 Ph.D. thesis (2009). 404

405

406

407

D. M. Chickering, A Transformational Characterization of Equivalent Bayesian Network Structures, UAI '95: Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence (Morgan Kaufmann, 1995), pp. 87-98.

M. Kalisch, M. Mächler, D. Colombo, M. H. Maathuis, P. Bühlmann, Causal inference using graphical 408 models with the r package pcalg. J. Stat. Softw. 47 , 1-26 (2012). 409

Appendix / supplemental material 410

411

A Preliminaries: connection and separation criteria

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

A.1 m -connection vs m' -connection criteria

An ancestral graph can be interpreted as encoding a set of conditional indepencence relations by a graphical criterion, called m -separation, based on the concept of m -connecting paths, which generalizes the separation criteria of Markov and Bayesian networks to ancestral graphs.

Definition 4. [ m -connecting path] A path π between X and Y is m -connecting given a (possibly empty) subset C ⊆ V (with X,Y / ∈ C ) if:

its non-collider(s) are not in C , and

ii) its collider(s) are in An G ( C ) .

Definition 5. [ m -separation criterion] The subsets A and B are said to be m -separated by C , noted A ⊥ m B | C , if there is no m -connecting path between any vertex in A and any vertex in B given C .

The probabilistic interpretation of ancestral graph is given by its global and pairwise Markov properties (which are equivalent [8]): if A and B are m -separated by C , then A and B are conditionally independent given C and ∀ X ∈ A and ∀ Y ∈ B , there is a probability distribution P faithful to G such that their conditional mutual information vanishes, i.e. I P ( X ; Y | C ) = 0 , also noted X ⊥ ⊥ P Y | C .

However, as discussed above, the proof of Theorem 1 will require to introduce a weaker m ′ -connection criterion defined below.

Definition 6. [ m ′ -connecting path] A path π between X and Y is m ′ -connecting given a subset C ⊆ V (with X,Y possibly in C ) if:

its non-collider(s) are not in C , and

ii)

its collider(s) are in

An

G

(

{

X,Y

} ∪

C

)

.

Note, in particular, that an m -connecting path is necessary an m ′ -connecting path but that the converse is not always true. For example, the path X → Z ←→ T ←→ Y in Fig. 1G (with Z → Y ) is an m ′ -connecting path given T (as Z ∈ An G ( { X,Y } ∪ T ) ) but not an m -connecting path given T (as Z / ∈ An G ( T ) ).

However, Richardson and Spirtes 2002 [8] have shown the following lemma,

Lemma 4. [Corollary 3.15 in [8]] In an ancestral graph G , there is a m ′ -connecting path µ between X and Y given C if and only if there is a (possibly different) m -connecting path π between X and Y given C .

Hence, Lemma 4 implies that m ′ -separation and m -separation criteria are in fact equivalent, as an absence of m ′ -connecting paths implies an absence of m -connecting paths and vice versa. This enables to reformulate the m -separation criterion above as,

Definition 7. [ m ′ -separation (and m -separation) criteria] The subsets A and B are said to be m ′ -separated (or m -separated) by C , if all paths from any X ∈ A to any Y ∈ B have either i)

a non-collider in

C

, or

The probabilistic interpretation of an ancestral graph is given by its (global) Markov property: if A and B are m -separated (or m ′ -separated) by C , then A and B are conditionally independent given C , noted as, A ⊥ m B | C .

A.2 ac -connecting paths and ac -connected subsets

Let us now recall the definition of ancestor collider connecting paths or ac -connecting paths , which is directly relevant to characterize the likelihood decomposition and Markov equivalent classes of ancestral graphs (Theorem 1). We give here a different yet equivalent definition of ac -connecting paths as defined in the main text (Definition 2) in order to underline the similarities and differencies with the notion of m ′ -connecting path (Definition 6).

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

Definition 8. [ ac -connecting path] A path π between X and Y is an ac -connecting path given a ⊆

Finally, we recall the notion of ac -connected subset for the decomposition of the likelihood of ancestral graphs (Theorem 1): A subset C ac -connected if ∀ X,Y ∈ C , there is an ac -connecting path between X and Y w.r.t. C .

B Proof of Theorem 1.

.

We will prove it by contradiction assuming that there exists a subset W ⊆ A , such that S = { X,Y } ∪ W is ac -connected. In particular, there should be an ac -connecting path between X and Y

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

confined to An G ( S ′ ) and thus to An G ( S ) ⊇ An G ( S ′ ) , which is an m ′ -connecting path between X and Y given A , contradicting the above hypothesis of m ′ -separation given A , i.e. I ( X ; Y | A ) = 0 . The use of m ′ -separation, i.e. the absence of m ′ -connecting paths with colliders in An G ( S ) rather than m -connecting paths with colliders in An G ( A ) , is necessary here, see Definitions 4 and 6. Hence, no ac -connected subset S ′ is included in cancelling combinations of multivariate information terms associated to pairwise conditional independence, I ( X ; Y | A ) = ∑ X,Y ∈ S ′ S ′ ⊆ S ( -1) | S ′ | I ( S ′ ) = 0 .

iii) Finally, we will show that the information terms which appear in multiple cancelling combinations from different pairwise independence constraints do not modify the multivariate information decomposition of the likelihood function of ancestral graphs, Eq. 12, as these shared/overlapping terms in fact all cancel through more global Markov independence relationships involving higher order (three or more points) vanishing multivariate information terms, such as I ( X ; Y ; Z | A ) = 0 .

This result requires to use an ordering of the nodes, X k glyph[follows] X j glyph[follows] X i , that is compatible with the directed edges of the ancestral graph assumed to have no undirected edges, i.e. X j / ∈ An ( X i ) if X j glyph[follows] X i . Under this ordering, higher order nodes X k glyph[follows] X i glyph[follows] X j can be a priori excluded from all separating sets A ij of pairs of lower order nodes, i.e. if I ( X i ; X j | A ij ) = 0 then X k / ∈ A ij .

glyph[negationslash]

In particular, the two pairwise conditional independence relations I ( X k ; X glyph[lscript] | A kglyph[lscript] ) = 0 , with X glyph[lscript] glyph[follows] X k , and I ( X i ; X j | A ij ) = 0 , with X j glyph[follows] X i , do not share any multivariate information terms, if X glyph[lscript] = X j . Indeed, as I ( X k ; X glyph[lscript] | A kglyph[lscript] ) contains all information terms including both X k and X glyph[lscript] as well as every subset (possibly empty) of A kglyph[lscript] , none of them includes X j if X glyph[lscript] glyph[follows] X j . Therefore I ( X k ; X glyph[lscript] | A kglyph[lscript] ) does not contain any information term of I ( X i ; X j | A ij ) which contains both X i and X j as well as every subset (possibly empty) of A ij . This property eliminates all multiple counting of multivariate informations terms shared if X glyph[lscript] = X j . Note that this result does not hold in general for ancestral graphs including undirected edges.

glyph[negationslash]

Hence, the issue of redundant multivariate information terms in the likelihood decomposition, Eq. 12, is related to the conditional independences of two or more pairs, { X i , X r } , { X j , X r } , ..., { X glyph[lscript] , X r } , sharing the same higher order node, X r . However, this situation also entails a more global Markov independence constraint between X r and { X i , X j , · · · , X glyph[lscript] } , given a separating set A , which can be decomposed into more local independence constraints using the chain rule and the decomposition rules of multivariate information (Eq. 9),

where all the conditional multivariate information terms vanish by induction due to the nonnegativity of (conditional) mutual information. In particular, the conditional multivariate information terms in the last expression, i.e. between X r and each subset of { X i , X j , · · · , X glyph[lscript] } given the separating set A , all vanish. This result can be readily extended to any subsets { X r , X s , · · · , X z } (conditionally) independent of { X i , X j , · · · , X glyph[lscript] } given a separating set A , i.e. I ( { X i , X j , · · · , X glyph[lscript] } ; { X r , X s , · · · , X z }| A ) = 0 . Hence, as the final conditional multivariate cross information terms of the decomposition all vanish while not sharing any subsets of variables, it proves the absence of redundancy and a global cancellation of nonac -connected subsets (from pairwise and higher order conditional independence relations) in the likelihood function of ancestral graphs without undirected edges, Eq. 12.

Hence, only ac -connected subsets effectively contribute to the cross-entropy of an ancestral graph with only directed and bidirected edges, Eq. 12. glyph[square]

C Factorization of the probability distribution of ancestral graphs 549

550

C.1 Factorization resulting from Theorem 1 and Proposition 3

551

552

553

Before presenting the factorization of the model distribution of ancestral graphs resulting from Theorem 1 and Proposition 3, it is instructive to obtain an equivalent factorization for Bayesian graphs, assuming a positive empirical distributions, p ( x 1 , · · · , x m ) = ∏ m i =1 p ( x i | x i -1 , · · · , x 1 ) > 0 ,

This leads to the following alternative expressions for the cross-entropy H ( p, q ) = 554 -∑ x p ( x ) log q ( x ) in terms of multivariate entropy and information, which only depend on the 555 empirical joint distribution p ( x ) , 556

557

558

559

560

561

562

563

564

565

where ∑ m i =1 I ( X i ; X i -1 \ Pa X i | Pa X i ) can be decomposed, using the chain rule and Eq. 11, into unconditional multivariate information terms, which exactly cancel all the multivariate information of the nonac -connected subsets of variables in the multivariate entropy decomposition, Eq. 6.

Note, however, that this result obtained for Bayesian networks requires an explicit factorization of the global model distribution, q ( x ) , in terms of the empirical distribution, p ( x ) , which is not known and presumably does not exist, in general, for ancestral graphs.

Alternatively, assuming that the empirical and model distributions are positive ( ∀ x , p ( x ) > 0 , q ( x ) > 0 ), it is always possible to factorize them into factors associated to each (cross) information term in the (cross) entropy decomposition, Eq. 6, as,

glyph[negationslash]

where all the marginal distributions over a subset of variables, e.g. q ( x i , x j , x k ) = ∑ glyph[lscript] = i,j,k q ( x ) or 566 p ( x i , x j , x k ) = ∑ glyph[lscript] = i,j,k p ( x ) , cancel two-by-two by construction. 567

glyph[negationslash]

This can be illustrated on a simple example of a two-collider path including one bidirected edge, 568 X → Z ←→ Y ← W (Fig. 1D), valid for q ( . ) and p ( . ) alike, 569

where all individual distribution marginals on subsets of variables, e.g. q ( x ) , q ( x, z ) , q ( x, z, y ) (or 570 p ( x ) , p ( x, z ) , p ( x, z, y ) ), cancel two-by-two by construction, except q ( x, z, y, w ) (or p ( x, z, y, w ) ). 571

In addition and only for the model distribution q ( . ) , all ratios in gray in Eq. 18 also cancel due to 572 Markov independence relations across nonac -connected subsets (see proof of Theorem 1). This 573 leaves a truncated factorization retaining all and only the ac -connected subsets of variables in the 574 graph, which we propose to estimate on empirical data by substituting the remaining q ( . ) terms by 575 their empirical counterparts p ( . ) , see Proposition 3. 576

This leads to the following global factorization for q ( . ) in terms of p ( . ) , 577

where the terms in gray have been passed to the lhs of Eq. 18 applied to p ( . ) . This ultimately 578 leads to the analog of the Bayesian Network factorization in Eq. 15 but for the two-collider path, 579 X → Z ←→ Y ← W (Fig. 1D), 580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

where the last three factors 'correct' the expression of p ( x, z, y, w ) for the three (conditional) independences entailed by the underlying graph, that is, X ⊥ W , Z ⊥ W | X , and X ⊥ Y | W .

C.2 Relation to the head-and-tail factorizations

The head-and-tail factorizations of the model distribution of an acyclic directed mixed graph, introduced by Richardson 2009 [6], enable the parametrization of the joint probability distribution with independent parameters for ancestrally closed subsets of vertices.

For instance, the head-and-tail factorizations of the simple two-collider path including one bidirected edge, X → Z ←→ Y ← W , introduced above, Fig. 1D, are [6],

Importantly, these head-and-tail factorizations imply additional relations such as q ( y | w ) = q ( y | x, w ) ( i.e. X ⊥ Y | W ) obtained by comparing the last two relations in Eq. 21 after marginalizing q ( x, z, y, w ) over z . However, such implicit conditional independence relations are not verified by the empirical distribution p ( . ) in general and prevent the estimation of the head-and-tail factorizations by substituting the rhs q ( . ) terms in Eq. 21 with their empirical counterparts p ( . ) , as in the case of Bayesian networks, Eq. 15.

Indeed, while the head-and-tail factorization relations, Eq. 21, obey the local and global Markov independence relations entailed by the graphical model, Fig. 1D, leading to the cancellation of all factors associated to nonac -connected subsets in gray in Eq. 18, the remaining head-and-tail factors cannot be readily estimated with the empirical distribution p ( . ) .

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

In particular, the cross-entropy of the two-collider path of interest, Fig. 1D, obtained with the headand-tail factorizations corresponds to 1 H ( p, q )= -∑ p ( x, z, y, w ) log q ( z, y | x, w ) q ( x ) q ( w ) . Then, estimating the q ( . ) terms with their p ( . ) counterparts leads to the cross-entropy of a Bayesian graph, Fig. 1E, with a different Markov equivalent class than the ancestral graph of interest, Fig. 1D. A similar discrepancy is obtained with a c-component factorization which leads to the cross-entropy of the Bayesian graph of Fig. 1E without edge X → Y , corresponding to a different Markov equivalence class than the previous two graphs, Figs. 1D & E.

These examples illustrate the difficulty to exploit the c-component or head-and-tail factorizations to estimate the likelihood of ancestral graphs including bidirected edge(s).

D Node and edge scores based on Normalized Maximum Likelihood criteria

Search-and-score methods based on likelihood estimates need to properly account for finite sample size, as cross-entropy minimization leads to ever more complex models, resulting in model overfitting for finite datasets. While BIC regularization is valid in the asymptotic limit of very large datasets, it tends to overestimate finite size corrections, leading to lower recall, in general. In order to better take into account finite sample size, we used instead the (universal) Normalized Maximum Likelihood (NML) criteria [42, 43, 38, 39], which amounts to normalizing the likelihood function over all possible datasets with the same number N of samples.

Node score . We first used the factorized Normalized Maximum Likelihood (fNML) complexity [38, 39] to define a local score for each node X i , which extends the decomposable likelihood of Bayesian graphs given each node's parents, Eq. 2, to all non-descendant neighbors, Pa ′ Xi ,

where n jk corresponds to the number of data points for which X i is in its k th state and its non619 descendant neighbors in their j th state, with n j = ∑ r i k n jk . The universal normalization constant C r n 620 is then computed by summing the numerator over all possible partitions of the n data points into a 621 maximum of r subsets, glyph[lscript] 1 + glyph[lscript] 2 + · · · + glyph[lscript] r = n with glyph[lscript] k glyph[greaterorequalslant] 0 , 622

which can in fact be computed in linear-time using the following recursion [38], 623

with C 1 n = 1 for all n and applying Eq. 30 below for r = 2 . However, for large n and r , C r n 624 computation tends to be numerically unstable, which can be circumvented by implementing the 625 recursion on parametric complexity ratios D r n = C r n / C r -1 n rather than parametric complexities 626 themselves [35] as, 627

for r glyph[greaterorequalslant] 3 , with C 1 n = 1 and C 2 n = D 2 n , which can be computed directly with the general formula, 628 Eq. 26, for r = 2 , 629

or its Szpankowski approximation for large n (needed for n > 1000 in practice) [44-46], 630

This leads to the following local score for each node X i , which is minimized over alternative combinations of non-descendant neighbors, Pa ′ Xi ⊆ Pa Xi ∪ Sp Xi ∪ Ne Xi , in the first step of the local search-and-score algorithm (step 1) detailed in the main text,

Edge scores . We then defined several edge scores to optimize the orientation of each edge, X Y , given its close surrounding vertices.

To this end, we first introduced a local score for node pairs which simply sums the node scores, Eq. 33, for each node. The resulting pair scores are listed in Table 2 for unconnected node pairs and for pairs of nodes connected by a directed edge, where Pa ′ X \ Y = Pa X ∪ Sp X \ Y and Pa ′ Y \ X = Pa Y ∪ Sp Y \ X with their corresponding combinations of levels, q y \ x and q x \ y .

glyph[negationslash]

glyph[negationslash]

Then, edge scores for directed edges, X → Y and Y → X , are defined w.r.t. to the edge removal 641 score, X Y , by substracting the pair scores of unconnected pairs to the pair scores of directed 642 edges, leading to the following edge orientation scores, 643

glyph[negationslash]

However, if r x = r y , the fNML complexities of these orientation scores are not identical for 644 Markov equivalent edge orientations between nodes sharing the same parents (or spouses) [47], 645 Pa ′ Y \ X = Pa ′ X \ Y = Pa ′ and q y \ x = q x \ y , despite sharing the same conditional mutual information, 646

This suggests to symmetrize the fNML complexities for edge orientation scores by averaging them 647 over each directed orientation, as for the conditional information in Eq. 36, leading to the proposed 648 fNML complexity for directed edges given in Table 1 in the main text. 649

631

632

633

634

635

636

637

638

639

640

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

For bidirected edges, the proposed local orientation score accounts for all ac -connected subsets in close vicinity of the bidirected edge, which concerns all subsets including either X and any combination (possibly void) of parents or spouses different from Y ( i.e. corresponding to the information contributions H ( X | Pa ′ X \ Y ) ) or Y and any combination of parents or spouses different from X ( i.e. corresponding to the information contributions H ( Y | Pa ′ Y \ X ) ) or, else, including both nodes X and Y plus any combination of their parents or spouses, corresponding to the following information contribution, -I ( X ; Y | Pa ′ XY ) , where Pa ′ XY = Pa ′ X \ Y ∪ Pa ′ Y \ X . This last term, -I ( X ; Y | Pa ′ XY ) , contains all the remaining information contributions once the bidirected orientation score is given relative to the edge removal score (Table 2) as for the two directed orientation scores, above. Finally, the symmetrized fNML complexity associated with a bidirected edge should be computed with the whole set of conditioning parents or spouses, Pa ′ XY , as indicated in Table 1. Note that this bidirected orientation score becomes also Markov equivalent to the two directed orientation scores, as required, when the nodes share the same parents and spouses, i.e. Pa ′ XY = Pa ′ Y \ X = Pa ′ X \ Y and q xy = q y \ x = q x \ y in Table 1.

E Toy models

Fig. 3 shows three simple ancestral models used to test MIIC_search&score orientation scores (Table 1) to effectively predict bidirected orientations when the end nodes do not share the same parents (Model 1), share some parents (Model 2) or when the bidirected edge is part of a longer than two-collider paths (Model 3).

The data is generated from the theoretical DAG using the rmvDAG function in the pcalg package [48]. Each node follows a normal distribution, and the data is discretized using bnlearn's discretize function using Hartemink's pairwise mutual information method [40]. For these toy models, the edge orientation scores are computed assuming the correct parents of each node.

The prediction of the edge orientation scores are summarized in Table 3 in % of replicates displaying directed edges (wrong) or bidirected edge (correct) as a function of increasing dataset size N .

Model 2

Model 3

Model 1

X1

X1

X1

X5

X2

X5

X2

X5

X6

X2

X6

X

X7

X

X

X6

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims of the paper are supported by the theoretical and experimental results shown in Figs. 1 & 2, respectively.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a Discussion & Limitation section at the end of the paper. The main limitation of the experimental results is the fact that we did not have sufficient time to perform many dataset replicates of the benchmark ancestral graphs. While the obtained statistics already support our main experimental results, we intend to perform more dataset replicates for the final version of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For the theoretical results (notably Theorem 1) we provide the full set of assumptions (section 2 and Appendix A)and a complete proof (Appendix B).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided the full description of the experiments run in the paper (sections 2 &3and Appendix D). The open-source code reproducing the experimental results presented in the paper will be provided with the camera-ready version of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not include a new code with the initial submission, as it is not yet properly packaged at submission time, but we definitely intend to release this open-source code including proper annotation and userguide with the final camera-ready version of the paper. MIIC and FCI open-source packages used for benchmark comparison are already published and available on public servers.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided the full description of the experiments run in the paper (sections 2 3 and Appendix D).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The 1-sigma error bars are plotted in Fig. 2. While these statistics already support our experimental results, we intend to perform more dataset replicates for the final version of the paper, which we did not have sufficient time to perform by the time of submission. This should reduce some error bars, in particular, those for the results displaying large error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resource used for all experiments is a simple laptop with intel i7 processors, 12 cores and 16 threads.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper does not use or produce sensitive data nor concern potentially harmful applications.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper does not use or produce sensitive data nor concern potentially harmful applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not use or produce sensitive data nor concern potentially harmful applications.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986

Answer: [Yes]

Justification: We have credited all previously published resources (including license details) used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not include a new code with the initial submission, as it is not yet properly packaged at submission time, but we definitely intend to release this open-source code including proper annotation and userguide with the final camera-ready version of the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a greedy structure learning algorithm in the context of ancestral graphs. The score function is based on the normalized likelihood. The score is decomposable using the concept of ac-components. Limited experimental results are presented as experimental evidence.\n\nSTRENGTHS:\n- The multivariate cross-entropy decomposition is explained in details, especially the link between its definition and its application in the context of ancestral graphs. Leveraging this framework allows a flexible and effective definition of scoring criteria for the class of graphical models.\n- The usage of ac-connecting paths and ac-connected subsets for the likelihood decomposition (theorem 1) is the main result of the paper and allows to map the structure of an ancestral graph to a specific estimand.\n\nWEAKNESSES:\n- In the case of ancestral graphs, it's crucial to understand the differences between edge patterns. I would suggest to refactor Figure 1 to improve readability, e.g. isolating different edge patterns combinations with horizontal lines.\n- The performance of the proposed solution is poor. For instance, Figure 2 most of the metrics overlap with existing solutions.\n- The experiments keep the number of samples fixed even is the number of the parameters of the model increase, reducing the information available for the biggest models. Moreover, a sample size of 100 for BARLEY (114,005 parameters) is rather unrealistic.\n\nQUESTIONS:\n- Are the limitations related to the cap of two-collider paths only? Are there any issues related to computational complexity? This was partially referenced in the attached form.\n- What was the reason of choosing only three models for benchmarks? It is unclear the actual advantage of such approach w.r.t. the existing FCI algorithm.\n- Why keeping the sample size range fixed even if the number of possible states increases dramatically? Observing 100 samples for BARLEY means that (ideally) each algorithm observe less than 1% of the state space, which limits the experimental evidence.\n\nLIMITATIONS:\n- The experimental evidence is poor: few reference models with rather sparse hyperparameters space exploration.\n- The search and score algorithm limits the number of possible exploration paths to the barely minimum.\n- It is unclear what is the actual computational burden of the proposed score.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
In this paper, the authors present a novel likelihood decomposition of ancestral graphs, which is based on a novel concept ac-connected subset. With this decomposition, the authors present an efficient hybrid causal discovery method. But it is worthy to note that in the implementation of the causal discovery method, they use the approximate local scores limited to the closed surrounding vertices of each node and edge, but not the exact likelihood score.

STRENGTHS:
1. The likelihood decomposition of ancestral graphs is very novel and could be useful in the future.
2. For me, I encourage the studies on the MAG/PAG learning methods. It is a hard and fundamental problem.
3. The paper is clearly written.

WEAKNESSES:
See questions.

QUESTIONS:
1. Could authors elaborate more on where Theorem 1 can be applied?

2. I am curious about how to find ac-connected subset, and what the computational complexity is.

3. One important thing that the readers may wonder is, what is the performance difference between the method based on approximate local scores limited to the closed surrounding vertices and the method with exact scores. Could the authors present more clues?

4. Could the authors present some efficiency comparisons empirically, as the authors say "to implement an efficient and reliable heuristic method applicable to more challenging graphical modes and large datasets". 

I am happy to increase my score if the questions are addressed.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper defines a score for ancestral graphs through an inclusion-exclusion expansion and implements it into a hybrid search approach.

STRENGTHS:
The likelihood decomposition (Theorem 1) is nice and there seems to be a slight improvement in the empirical testing.

WEAKNESSES:
There are close overlaps to published and preprint works by Hu and Evans, so it is hard to assess the novelty here. For example Theorem 1 (in another form) is from reference 14. 

Then in the simulation studies, relevant methods from the references are excluded, even though the greedy search approaches (like references 13 and 32 for example) are not exact, but also heuristic. These (and similar) would need to be included in the benchmarking to judge the importance of the work.

QUESTIONS:
Can the authors more clearly delineate what is novel and different here as compared to the other recent works of search-and-score for ancestral graphs?

LIMITATIONS:
The limitation on the depth of the inclusion/exclusion is discussed in the limitation section, but maybe this wasn't so clear throughout the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a greedy search-and-score algorithm for causal structure discovery in ancestral graphs which allow for both directed and bidirected edges.The paper provides an explicit decomposition of the likelihood function of ancestral graphs\nin terms of multivariate cross-information over relevant ‘ac-connected’ subsets of variables which is related to head-and-tail factorization developed in prior work. In the main theoretical result authors show that cross-entropy and the likelihood function of an ancestral graph can be decomposed in terms of multivariate cross-information for ac-connected components. As a corollary this implies that two ancestral graphs are Markov equivalent if and only if they have the same ac-connected subsets of vertices.\nAuthors use the likelihood function decomposition in terms of cross-information of ac-connected componenetsto propose a search-and score algorithm that computes scores for nodes and edges and probes orientations of edges to minimize the score.\n\nSTRENGTHS:\nThe paper proposes a new empirical algorithm for causal structure discovery in ancestral graphs. The practical algorithm is motivated by a theoretical decomposition of likelihood function in terms of multivariate cross-information summed over ac-connected components. The developed algorithm is scalable to graphs with several dozens of vertices and links (thousands of unknown parameters)\n\nThe authors provide experimental results on both synthetic datasets and challenging benchmarks from the bnlearn repository, showing that developed algorithm typically has higher precision and similar recall to prior MIIC algorithm.\n\nWEAKNESSES:\nThe proposed algorithm seems to be only empirical and lacks theoretical guarantees and relies on MIIC as a base of the algorithm. To my understanding, the paper does not provide a guarantee that the algorithm always converges in a reasonable number of steps (even if not to a global minimum). Multivariate cross-information over a large set of variables may be very sensitive to noise, which I think is a weakness compared to algorithms that use cross-information only over small number of (e.g. triples) of variables.\n\nAlso as the authors point out Theorem 1 is equivalent to the decomposition established in prior work, which makes novelty of the main theorem somewhat questionable.\n\nI also think that it will be beneficial to the reader if the connection between the theoretical results in Theorem 1 and the proposed algorithm are explained a bit more deeply with a more clear connection of how that theorem is being used.\n\nQUESTIONS:\n1) What is the main Novelty of Theorem 1 compared to [14]? Can the theoretical motivation for the proposed search-and-score algorithm be deduced from the results in [14]?\n2) Do you have convergence guarantees for the algorithm? In other words, how many edge orientations need to be probed before convergence in step 1? How many loops over nodes and edges are needed until convergence? \n4) Is there any class of distributions or graphs for which algorithm is guaranteed to converge to a true hidden causal graph?\n5) Did you perform any experiment to understand how sensitive is the proposed algorithm to the noise?\n6) In the first step of the algorithm does it matter in which order vertices are processed? do you select any specific order?\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}