Title: Learning bounded-degree polytrees with samples
Venue: NeurIPS 2023
Accepted: no
Keywords: Bayesian networks

finite samples

polytrees

learning
Abstract: We establish finite-sample guarantees for efficient proper learning of bounded-degree polytrees, a rich class of high-dimensional probability distributions and a subclass of Bayesian networks, a widely-studied type of graphical models. Very recently, Bhattacharyya et al. [2021] obtained finite-sample guarantees for recovering tree-structured Bayesian networks, i.e., 1-polytrees. We considerably extend their results by providing an efficient algorithm which learns d-polytrees in polynomial time and sample complexity when the in-degree d is constant, provided that the underlying undirected graph (skeleton) is known. We complement our algorithm with an information-theoretic lower bound, showing that the dependence of our sample complexity is nearly tight in both the dimension and target accuracy parameters.

1

Abstract

We establish finite-sample guarantees for efficient proper learning of boundeddegree polytrees , a rich class of high-dimensional probability distributions and a subclass of Bayesian networks, a widely-studied type of graphical models. Very recently, Bhattacharyya et al. [2021] obtained finite-sample guarantees for recovering tree-structured Bayesian networks, i.e., 1-polytrees. We considerably extend their results by providing an efficient algorithm which learns d -polytrees in polynomial time and sample complexity when the in-degree d is constant, provided that the underlying undirected graph (skeleton) is known. We complement our algorithm with an information-theoretic lower bound, showing that the dependence of our sample complexity is nearly tight in both the dimension and target accuracy parameters.

1 Introduction

Distribution learning, or density estimation, is the task of obtaining a good estimate of some unknown underlying probability distribution P from observational samples. Understanding which classes of distributions could be or could not be learnt efficiently is a fundamental problem in both computer science and statistics, where efficiency is measured in terms of sample (data) and computational (time) complexities.

Bayesian networks (or Bayes nets in short) represent a class of high-dimensional distributions that can be explicitly described by how each variable is be generated sequentially in a directed fashion. Being interpretable, Bayes nets have been used to model beliefs in a wide variety of domains (e.g. see [Jensen and Nielsen, 2007, Koller and Friedman, 2009] and references therein). A fundamental problem in computational learning theory is to identify families of Bayes nets which can be learned efficiently from observational data.

Formally, a Bayes net is a probability distribution P , defined over some directed acyclic graphs (DAG) 23 G = ( V, E ) on | V | = n nodes that factorizes according to G (i.e. Markov with respect to G ) in the 24 following sense: P ( v 1 , . . . , v n ) = ∏ v 1 ,...,v n P ( v | π ( v )) , where π ( v ) ⊆ V are the parents of v in G . 25 While it is well-known that given the DAG (structure) of a Bayes net, there exists sample-efficient 26 algorithms that output good hypotheses [Dasgupta, 1997, Bhattacharyya et al., 2020], there is no 27 known computationally efficient algorithms for obtaining the DAG of a Bayes net. In fact, it has long 28 been understood that Bayes net structure learning is computationally expensive, in various general 29 settings [Chickering et al., 2004]. However, these hardness results fall short when the goal is learning 30 the distribution P in the probabilistically approximately correct (PAC) [Valiant, 1984] sense (with 31 respect to, say, KL divergence or total variation distance), rather than trying to recover an exact graph 32 from the Markov equivalence class of P . 33

Polytrees are a subclass of Bayesian networks where the undirected graph underlying the DAG is 34

a forest, i.e., there is no cycle for the undirected version of the DAG; a polytree with maximum 35

in-degree d is also known as a d -polytree. With an infinite number of samples, one can recover the 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

DAG of a non-degenerate polytree in the equivalence class with the Chow-Liu algorithm [Chow and Liu, 1968] and some additional conditional independence tests [Rebane and Pearl, 1988]. However, this algorithm does not work in the finite sample regime. The only known result for learning polytrees with finite sample guarantees is for 1-polytrees [Bhattacharyya et al., 2021]. Furthermore, in the agnostic setting, when the goal is to find the closest polytree distribution to an arbitrary distribution P , the learning problem becomes NP-hard [Dasgupta, 1999].

In this work, we investigate what happens when the given distribution is a d -polytree, for d > 1 . Are d -polytrees computationally hard to learn in the realizable PAC-learning setting? One motivation for studying polytrees is due to a recent work of Gao and Aragam [2021] which shows that polytrees are easier to learn than general Bayes nets due to the underlying graph being a tree, allowing typical causal assumptions such as faithfulness to be dropped when designing efficient learning algorithms.

Contributions. Our main contribution is a sample-efficient algorithm for proper Bayes net learning in the realizable setting, when provided with the ground truth skeleton (i.e., the underlying forest). Crucially, our result does not require any distributional assumptions such as strong faithfulness, etc.

Theorem 1. There exists an algorithm which, given m samples from a polytree P over Σ n , accuracy parameter ε > 0 , failure probability δ , as well as its maximum in-degree d and the explicit description of the ground truth skeleton of P , outputs a d -polytree ˆ P such that d KL ( P ∥ ˆ P ) ≤ ε with success probability at least 1 -δ , as long as

Moreover, the algorithm runs in time polynomial in m , | Σ | d , and n d .

We remark that our result holds when even given only an upper bound on the true in-degree d . In particular, our result provides, for constant | Σ | , d , an upper bound of ˜ O ( n/ε ) on the sample complexity of learning O (1) -polytrees. Note that this dependence on the dimension n and the accuracy parameter ε are optimal, up to logarithmic factors: indeed, we establish in Theorem 15 an Ω( n/ε ) sample complexity lower bound for this question, even for d = 2 and | Σ | = 2 . 1

We also state sufficient conditions on the distribution that enable recovery of the ground truth skeleton. Informally, we require that the data processing inequality hold in a strong sense with respect to the edges in the skeleton graph. Under these conditions, combining with our main result in Theorem 1, we obtain a polynomial-time PAC algorithm to learn bounded-degree polytrees from samples.

Other related work. Structure learning of Bayesian networks is an old problem in machine learning and statistics that has been intensively studied; see, for example, Chapter 18 of Koller and Friedman [2009]. Many of the early approaches required faithfulness, a condition which permits learning of the Markov equivalence class, e.g. Spirtes and Glymour [1991], Chickering [2002], Friedman et al. [2013]. Finite sample complexity of such algorithms assuming faithfulness-like conditions has also been studied, e.g. Friedman and Yakhini [1996]. An alternate line of more modern work has considered various other distributional assumptions that permits for efficient learning, e.g., Chickering and Meek [2002], Hoyer et al. [2008], Shimizu et al. [2006], Peters and Bühlmann [2014], Ghoshal and Honorio [2017], Park and Raskutti [2017], Aragam et al. [2019], with the latter three also showing analyzing finite sample complexity. Specifically for polytrees, Rebane and Pearl [1988], Geiger et al. [1990] studied recovery of the DAG for polytrees under the infinite sample regime. Gao and Aragam [2021] studied the more general problem of learning Bayes nets, and their sufficient conditions simplified in the setting of polytrees. Their approach emphasize more on the exact recovery, and thus the sample complexity has to depend on the minimum gap of some key mutual information terms. In contrast, we allow the algorithm to make mistakes when certain mutual information terms are too small to detect for the given sample complexity budget and achieve a PAC-type guarantee. As such, once the underlying skeleton is discovered, our sample complexity only depends on the d, n, ε and not on any distributional parameters.

There are existing works on Bayes net learning with tight bounds in total variation distance, with a focus on sample complexity (and not necessarily computational efficiency); for instance, [Canonne et al., 2020]. Acharya et al. [2018] consider the problem of learning (in TV distance) a bounded-degree causal Bayesian network from interventions, assuming the underlying DAG is known.

Outline of paper. We begin with some preliminary notions and related work in Section 2. Section 3 87 then shows how to recover a polytree close in KL divergence, assuming knowledge of the skeleton 88 and maximum in-degree. Section 4 gives sufficient conditions to recover the underlying skeleton from 89 samples, while Section 5 provides our sample complexity lower bound. We conclude in Section 6 90 with some open directions and defer some full proofs to the appendix. 91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

2 Preliminaries and tools from previous work

2.1 Preliminary notions and notation

We write the disjoint union as ˙ ∪ . For any set A , let | A | denotes its size. We use hats to denote estimated quantities, e.g., ˆ I ( X ; Y ) will be the estimated mutual information of I ( X ; Y ) . We employ the standard asymptotic notation O ( · ) , Ω( · ) Θ( · ) , and write ˜ O ( · ) to omit polylogarithmic factors. Throughout, we identify probability distributions over discrete sets with their probability mass functions (pmf). We use d ∗ to denote the true maximum in-degree of the underlying polytree.

2.2 Probability distribution definitions

Webegin by defining KL divergence and squared Hellinger distances for a pair of discrete distributions with the same support.

Definition 2 (KL divergence and squared Hellinger distance) . For distributions P, Q defined on the same discrete support X , their KL divergence and squared Hellinger distances are defined as d KL ( P ∥ Q ) = ∑ x ∈X P ( x ) log P ( x ) Q ( x ) and d 2 H ( P, Q ) = 1 -∑ x ∈X √ P ( x ) · Q ( x ) respectively.

Abusing notation, for a distribution P on variables X = { X 1 , . . . , X n } , we write P S to mean the projection of P to the subset of variables S ⊆ X and P G to mean the projection of P onto a graph G . More specifically, we have P G ( x 1 , . . . , x n ) = ∏ x ∈ X P ( x | π G ( x )) where π G ( x ) are the parents of x in G . Note that P G is the closest distribution 2 on G to P in d KL , i.e. P G = argmin Q ∈ G d KL ( P ∥ Q ) . By Chow and Liu [1968], we also know that

where H is the entropy function. Note that only the first term depends on the graph structure of G .

By maximizing the sum of mutual information (the negation of the first term in (1)), we can obtain an ε -approximated graph G such that d KL ( P ∥ P G ) ≤ ε . In the case of tree-structured distributions, this can be efficiently solved by using any maximum spanning tree algorithm; a natural generalization to bounded degree bayes nets remains open due to the hardness of solving the underlying optimization problem [Höffgen, 1993]. If any valid topological ordering of the target Bayes net P is present, then an efficient greedy approach is able to solve the problem.

Definition 3 ((Conditional) Mutual Information) . Given a distribution P , the mutual information of two random variables X and Y , supported on X and Y respectively, is defined as

Conditioning on a third random variable Z , supported on Z , the conditional mutual information is defined as:

By adapting a known testing result from [Bhattacharyya et al., 2021, Theorem 1.3], we can obtain the following corollary, which we will use. We provide the full derivation in the supplementary materials.

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

Corollary 4 (Conditional Mutual Information Tester, adapted from [Bhattacharyya et al., 2021, Theorem 1.3]) . Fix any ε > 0 . Let ( X,Y,Z ) be three random variables over Σ X , Σ Y , Σ Z respectively. Given the empirical distribution ( ˆ X, ˆ Y , ˆ Z ) over a size N sample of ( X,Y,Z ) , there exists a universal constant 0 < C < 1 so that for any

the following statements hold with probability 1 -δ : (1) If I ( X ; Y | Z ) = 0 , then ˆ I ( X ; Y | Z ) < C · ε . (2) If ˆ I ( X ; Y | Z ) ≥ C · ε , then I ( X ; Y | Z ) > 0 . (3) If ˆ I ( X ; Y | Z ) ≤ C · ε , then I ( X ; Y | Z ) < ε . Unconditional statements involving I ( X ; Y ) and ˆ I ( X ; Y ) hold similarly by choosing | Σ Z | = 1 .

2.3 Graph definitions

Let G = ( V, E ) be a graph on | V | = n vertices and | E | nodes where adjacencies are denoted with dashes, e.g. u -v . For any vertex v ∈ V , we use N ( v ) ⊆ V \ { v } to denote the neighbors of v and d ( v ) = | N ( v ) | to denote v 's degree. An undirected cycle is a sequence of k ≥ 3 vertices such that v 1 -v 2 -. . . -v k -v 1 . For any subset E ′ ⊆ E of edges, we say that the graph H = ( V, E ′ ) is the edge-induced subgraph of G with respect to E ′ .

For oriented graphs, we use arrows to denote directed edges, e.g. u → v . We denote π ( v ) to denote the parents of v and d in ( v ) to denote v 's incoming degree. An interesting directed subgraph on three vertices is the v-structure, where u → v ← w and u \ -w ; we say that v is the center of the v-structure. In this work, we study a generalized higher-degree version of v-structures: we define the notion of degℓ v-structure as a node v with ℓ ≥ 2 parents u 1 , u 2 . . . , u ℓ . We say that a degℓ v-structure is said to be ε -strong if we can reliably identify them in the finite sample regime. In our context, it means that for all k ∈ [ ℓ ] , I ( u k ; { u 1 , u 2 . . . , u ℓ } \ u k | v ) ≥ C · ε , for the universal constant 0 < C < 1 appearing in Corollary 4. A directed acyclic graph (DAG) is a fully oriented graph without any directed cycles (a sequence of k ≥ 3 vertices such that v 1 → v 2 → . . . → v k → v 1 ) and are commonly used to represent the conditional dependencies of a Bayes net.

For any partially directed graph, an acyclic completion or consistent extension refers to an assignment of edge directions to unoriented edges such that the resulting fully directed graph has no directed cycles; we say that a DAG G is consistent with a partially directed graph H if G is an acyclic completion of H . Meek rules are a set of 4 edge orientation rules that are sound and complete with respect to any given set of arcs that has a consistent DAG extension Meek [1995]. Given any edge orientation information, one can always repeatedly apply Meek rules till a fixed point to maximize the number of oriented arcs. One particular orientation rule (Meek R 1 ) orients b → c whenever a partially oriented graph has the configuration a → b -c and a \ -c so as to avoid forming a new v-structure of the form a → b ← c . In the same spirit, we define Meek R 1( d ∗ ) to orient all incident unoriented edges away from v whenever v already has d ∗ parents in a partially oriented graph.

The skeleton skel( G ) of a graph G refers to the resulting undirected graph after unorienting all edges in G , e.g. see Fig. 1. A graph G is a polytree if skel( G ) is a forest. For d ≥ 1 , a polytree G is a d -polytree if all vertices in G have at most d parents. Without loss of generality, by picking the minimal d , we may assume that d -polytrees have a vertex with d parents. When we freely orient a forest, we pick arbitrary root nodes in the connected components and orient to form a 1-polytree.

3 Recovering a good orientation given a skeleton and degree bound

In this section, we describe and analyze an algorithm for estimating a probability distribution P that is defined on a d ∗ -polytree G ∗ . We assume that we are given skel( G ∗ ) and d as input.

Note that for some distributions there could be more than one ground truth graph, e.g. when the Markov equivalence class has multiple graphs. In such situations, for analysis purposes, we are free to choose any graph that P is Markov with respect to. As the mutual information scores 3 are the same for any graphs that P is Markov with respect to, the choice of G ∗ does not matter here.

171

172

173

174

175

182

183

184

185

186

d

b

a

c

f

e

g

h

i

j

(a)

G

∗

d

b

a

c

f

e

g

h

i

j

(b)

skel(

G

∗

)

3.1 Algorithm

At any point in the algorithm, let us define the following sets. Let N ( v ) be the set of all neighbors of v in skel( G ∗ ) . Let N in ( v ) ⊆ N ( v ) be the current set of incoming neighbors of v . Let N out ( v ) ⊆ N ( v ) be the current set of outgoing neighbors of v . Let N un ( v ) ⊆ N ( v ) be the current set of unoriented neighbors of v . That is,

Algorithm 1 Algorithm for known skeleton and max in-degree.

Input : Skeleton skel( G ∗ ) = ( V, E ) , max in-degree d ∗ , threshold ε > 0 , universal constant C Output : A complete orientation of skel( G ∗ )

1: Run Phase 1: Orient strong v-structures (Algorithm 3)

2: Run Phase 2: Local search and Meek R 1( d ∗ ) (Algorithm 4)

▷ O ( n d ∗ ) time

▷ O ( n 3 ) time

3: Run Phase 3: Freely orient remaining unoriented edges (Algorithm 5) ▷ O ( n ) time via DFS

4: return ˆ G

There are three phases to our algorithm. In Phase 1, we orient strong v-structures. In Phase 2, we 176 locally check if an edge is forced to orient one way or another to avoid incurring too much error. In 177 Phase 3, we orient the remaining unoriented edges as a 1-polytree. Since the remaining edges were 178 not forced, we may orient the remaining edges in an arbitrary direction (while not incurring 'too 179 much error') as long as the final incoming degrees of any vertex does not increase by more than 1. 180 Subroutine Orient (Algorithm 2) performs the necessary updates when we orient u -v to u → v . 181

Algorithm 2 Orient : Subroutine to orient edges

Input : Vertices u and v where u -v is currently unoriented

1: Orient u -v as u → v .

2: Update N in ( v ) to N in ( v ) ∪ { u } and N un ( v ) to N un ( v ) \ { u } .

3: Update N out ( u ) to N out ( u ) ∪ { v } and N un ( u ) to N un ( u ) \ { v } .

3.2 Analysis

Observe that we perform O ( n d ∗ ) (conditional) mutual information tests in Algorithm 1. The following lemma (Lemma 5) ensures us that all our tests will behave as expected with sufficient samples. As such, in the rest of our analysis, we analyze under the assumption that our tests are correct.

Lemma 5. Suppose all variables in the Bayesian network has alphabet Σ . For ε ′ > 0 , with

empirical samples, O ( n d ∗ ) statements of the following forms, where X and Y are variable sets of 187 size | X ˙ ∪ Y | ≤ d and Z is possibly ∅ , all succeed with probability at least 1 -δ : 188

193

194

195

196

197

198

Recall that π ( v ) is the set of true parents of v in G ∗ . Let H be the forest induced by the remaining unoriented edges after phase 2. Let ˆ G be returned graph of the algorithm 1. Let us denote the final N in ( v ) as π in ( v ) at the end of Phase 2, just before freely orienting, i.e. the vertices pointing into v in ˆ G \ H . Let π un ( v ) = π ( v ) \ π in ( v ) be the set of ground truth parents that are not identified in Phase 1. Since the algorithm does not make mistakes for orientations in ˆ G \ H (Lemma 6), all edges of in π un ( v ) will be unoriented.

199

200

201

202

203

204

205

206

207

208

209

210

211

Lemma 6. Any oriented arc in ˆ G \ H is a ground truth orientation. That is, any vertex parent set in ˆ G \ H is a subset of π ( v ) , i.e. π in ( v ) ⊆ π ( v ) , and N in ( v ) at any time during the algorithm will have N in ( v ) ⊆ π in ( v ) .

d

b

a

c

f

e

g

h

i

j

Figure 2: Suppose we have the following partially oriented graph in the execution of Algorithm 4 (after Phase 1). Since N in ( d ) = { a, b } , we will check the edge orientations of c -d and f -d . Since I ( f ; { a, b } | d ) = 0 , we will have ˆ I ( f ; { a, b } | d ) ≤ ε , so we will not erroneously orient f → d . Meanwhile, I ( c ; { a, b } ) = 0 , we will have ˆ I ( c ; { a, b } ) ≤ ε , so we will not erroneously orient d → c .

Let ˆ π ( v ) be the proposed parents of v output by Algorithm 1. The KL divergence between the true distribution and our output distribution is essentially ∑ v ∈ V I ( v ; π ( v )) -∑ v ∈ V I ( v ; ˆ π ( v )) as the structure independent terms will cancel out.

To get a bound on the KL divergence, we will upper bound ∑ v ∈ V I ( v ; π ( v )) and lower bound ∑ v ∈ V I ( v ; ˆ π ( v )) . To upper bound I ( v ; π ( v )) in terms of π in ( v ) ⊆ π ( v ) and I ( v ; u ) for u ∈ π un ( v ) , we use Lemma 8 which relies on repeated applications of Lemma 7. To lower bound ∑ v ∈ V I ( v ; ˆ π ( v )) , we use Lemma 9.

̸

Lemma 7. Fix any vertex v , any S ⊆ π un ( v ) , and any S ′ ⊆ π in ( v ) . If S = ∅ , then there exists a vertex u ∈ S ∪ S ′ with

Lemma 8. For any vertex v with π in ( v ) , we can show that

Algorithm 3 Phase 1: Orient strong v-structures

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

4 Skeleton assumption

In this section, we present a set of sufficient assumptions (Assumption 11) under which the Chow-Liu 229 algorithm will recover the true skeleton even while with finite samples. 230

Algorithm 4 Phase 2: Local search and Meek R 1( d ∗ )

̸

Algorithm 5 Phase 3: Freely orient remaining unoriented edges

1: Let H be the forest induced by the remaining unoriented edges.

2: Freely orient H as a 1-polytree (i.e. maximum in-degree in H is 1).

3: Let ˆ G be the combination of the oriented H and the previously oriented arcs.

4: return ˆ G

In Phase 3, we increase the incoming edges to any vertex by at most one. The following lemma tells us that we lose at most 4 an additive ε error per vertex.

Lemma 9. Consider an arbitrary vertex v with π in ( v ) at the start of Phase 3. If Phase 3 orients u → v for some u -v ∈ H , then

By using Lemma 8 and Lemma 9, we can show our desired KL divergence bound (Lemma 10).

Lemma 10. Let π ( v ) be the true parents of v . Let ˆ π ( v ) be the proposed parents of v output by our algorithm. Then,

With these results in hand, we are ready to establish our main theorem:

Proof of Theorem 1. We first combine Lemma 10 and Lemma 5 with ε ′ = ε 2 n · ( d ∗ +1) in order to obtain an orientation ˆ G which is close to G ∗ . Now, recall that there exist efficient algorithms for estimating the parameters of a Bayes net with in-degreed (note that this includes d -polytrees) P once a close-enough graph ˆ G is recovered [Dasgupta, 1997, Bhattacharyya et al., 2020], with sample complexity ˜ O ( | Σ | d n/ε ) . Denote the final output ˆ P ˆ G , a distribution that is estimated using the conditional probabilities implied by ˆ G . One can bound the KL divergences as follows:

Thus, we see that

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

d

b

a

c

f

e

g

h

i

j

(a) Before final phase

d

b

a

c

f

e

g

h

i

j

(b)

ˆ

G

under an arbitrary orienta-

tion of

H

; see Fig. 4 for more.

d

b

a

c

f

e

g

h

i

j

(c)

ˆ

G

\

H

d

b

a

c

f

e

g

h

i

j

(a)

c

as the root

d

b

a

c

f

e

g

h

i

j

(b)

d

as the root

d

b

a

c

f

e

g

h

i

j

(c)

f

as the root

d

b

a

c

f

e

g

h

i

j

(d)

e

as the root

d

b

a

c

f

e

g

h

i

j

(e)

h

as the root

Assumption 11. For any given distribution P , there exists a constant ε P > 0 such that: (1) For every pair of nodes u and v , if there exists a path u -··· -v of length greater than 2 in G ∗ , then then I ( u ; v ) + 3 · ε P ≤ I ( a ; b ) for every pair of adjacent vertices a -b in the path. (2) For every pair of directly connected nodes a -b in G ∗ , I ( a ; b ) ≥ 3 · ε P .

Suppose there is a large enough gap of ε P between edges in G ∗ and edges outside of G ∗ . Then, with O (1 /ε 2 P ) samples, each estimated mutual information ˆ I ( a ; b ) will be sufficiently close to the true mutual information I ( a ; b ) . Thus, running the Chow-Liu algorithm (which is essentially maximum spanning tree on the estimated mutual information on each pair of vertices) recovers skel( G ∗ ) .

Lemma 12. Under Assumption 11, running the Chow-Liu algorithm on the m -sample empirical estimates { ˆ I ( u ; v ) } u,v ∈ V recovers a ground truth skeleton with high probability when m ≥ Ω( log n ε 2 P ) .

Combining Lemma 12 with our algorithm Algorithm 1, one can learn a polytree that is ε -close in KL with ˜ O ( max { log( n ) ε 2 P , 2 d · n ε }) samples, where ε P depends on the distribution P .

5 Lower bound

In this section, we show that Ω( n/ε ) samples are necessary even when a known skeleton is provided . For constant in-degree d , this shows that our proposed algorithm in Section 3 is sample-optimal up to logarithmic factors.

We first begin by showing a lower bound of Ω(1 /ε ) on a graph with three vertices, even when the skeleton is given. Let G 1 be X → Z → Y and G 2 be X → Z ← Y , such that skel( G 1 ) = skel( G 2 ) is X -Z -Y . Now define P 1 and P 2 as follows:

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

The intuition is that we keep the edge X → Z 'roughly the same' and tweak the edge Y -Z between the distributions. We define P i,G as projecting P i onto G . One can check that the following holds (see Supplemental for the detailed calculations):

Lemma 13. Let G 1 be X → Z → Y and G 2 be X → Z ← Y , such that skel( G 1 ) = skel( G 2 ) is X -Z -Y . With respect to Eq. (3) , we have the following:

d 2 H ( P 1 , P 2 ) ∈ O ( ε )

Our hardness result (Lemma 14) is obtained by reducing the problem of finding an ε -close graph orientation of X -Z -Y to the problem of testing whether the samples are drawn from P 1 or P 2 : To ensure ε -closeness in the graph orientation, one has to correctly determine whether the samples come from P 1 or P 2 and then pick G 1 or G 2 respectively. However, it is well-known that distinguishing two distributions whose squared Hellinger distance is ε requires Ω(1 /ε ) samples (see, e.g., [Bar-Yossef, 2002, Theorem 4.7]).

Lemma14. Even when given skel( G ∗ ) , it takes Ω(1 /ε ) samples to learn an ε -close graph orientation of G ∗ for distributions on { 0 , 1 } 3 .

Using the above construction as a gadget, we can obtain a dependency on n in our lower bound by constructing n/ 3 independent copies of the above gadget, à la proof strategy of Bhattacharyya et al. [2021, Theorem 7.6]. For some constant c > 0 , we know that a constant 1 /c fraction of the gadgets will incur an error or more than ε/n if less than cn/ε samples are used. The desired result then follows from the tensorization of KL divergence, i.e., d KL ( ∏ i P i ∥ ∏ i Q i ) = ∑ i d KL ( P i ∥ Q i ) .

Theorem 15. Even when given skel( G ∗ ) , it takes Ω( n/ε ) samples to learn an ε -close graph orientation of G ∗ for distributions on { 0 , 1 } n .

6 Conclusion

In this work, we studied the problem of estimating a distribution defined on a d -polytree P with graph structure G ∗ using finite observational samples. We designed and analyzed an efficient algorithm that produces an estimate ˆ P such that d KL ( P ∥ ˆ P ) ≤ ε assuming access to skel( G ∗ ) and d . The skeleton skel( G ∗ ) is recoverable under Assumption 11 and we show that there is an inherent hardness in the learning problem even under the assumption that skel( G ∗ ) is given. For constant d , our hardness result shows that our proposed algorithm is sample-optimal up to logarithmic factors.

An interesting open question is whether one can extend the hardness result to arbitrary d ≥ 1 , or design more efficient learning algorithms for d -polytrees.

References

Acharya, A. Bhattacharyya, C. Daskalakis, and S. Kandasamy. Learning and testing causal models with interventions. In NeurIPS , pages 9469-9481, 2018.

Aragam, A. Amini, and Q. Zhou. Globally optimal score-based learning of directed acyclic graphs in high-dimensions. Advances in Neural Information Processing Systems , 32, 2019.

Bar-Yossef. The Complexity of Massive Data Set Computations . PhD thesis, UC Berkeley, 2002. Adviser: Christos Papadimitriou. Available at http://webee.technion.ac.il/people/ zivby/index_files/Page1489.html .

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

Bhattacharyya, S. Gayen, K. S. Meel, and N. V. Vinodchandran. Efficient distance approximation for structured high-dimensional distributions via learning. In NeurIPS , 2020.

Bhattacharyya, S. Gayen, E. Price, and N. V. Vinodchandran. Near-optimal learning of treestructured distributions by Chow-Liu. In STOC '21-Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 147-160. ACM, New York, 2021. doi: 10.1145/3406325.3451066. URL https://doi.org/10.1145/3406325.3451066 .

L. Canonne, I. Diakonikolas, D. M. Kane, and A. Stewart. Testing bayesian networks. IEEE Trans. Inf. Theory , 66(5):3132-3170, 2020. Preprint available at arXiv:1612.03156.

M. Chickering. Optimal structure identification with greedy search. Journal of machine learning research , 3(Nov):507-554, 2002.

M. Chickering and C. Meek. Finding optimal bayesian networks. In Proceedings of the Eighteenth Annual Conference on Uncertainty in Artificial Intelligence (UAI 02) , pages 94-102, 2002.

M. Chickering, D. Heckerman, and C. Meek. Large-sample learning of bayesian networks is np-hard. J. Mach. Learn. Res. , 5:1287-1330, 2004.

K. Chow and C. N. Liu. Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theory , 14(3):462-467, 1968.

Dasgupta. The sample complexity of learning fixed-structure bayesian networks. Mach. Learn. , 29(2-3):165-180, 1997.

Dasgupta. Learning polytrees. In UAI , pages 134-141. Morgan Kaufmann, 1999.

Friedman and Z. Yakhini. On the sample complexity of learning bayesian networks. In Proceedings of the Twelfth Annual Conference on Uncertainty in Articial Intelligence (UAI { 96 } ), San Francisco, CA , pages 274-282, 1996.

Friedman, I. Nachman, and D. Pe'er. Learning bayesian network structure from massive datasets: The" sparse candidate" algorithm. arXiv preprint arXiv:1301.6696 , 2013.

Gao and B. Aragam. Efficient bayesian network structure learning via local markov boundary search. Advances in Neural Information Processing Systems , 34:4301-4313, 2021.

Geiger, A. Paz, and J. Pearl. Learning causal trees from dependence information. In AAAI , pages 770-776. AAAI Press / The MIT Press, 1990.

Ghoshal and J. Honorio. Learning identifiable gaussian bayesian networks in polynomial time and sample complexity. Advances in Neural Information Processing Systems , 30, 2017.

K.-U. Höffgen. Learning and robust learning of product distributions. In Proceedings of the sixth annual conference on Computational learning theory , pages 77-83, 1993.

Hoyer, D. Janzing, J. M. Mooij, J. Peters, and B. Schölkopf. Nonlinear causal discovery with additive noise models. Advances in Neural Information Processing Systems , 21, 2008.

V. Jensen and T. D. Nielsen. Bayesian networks and decision graphs , volume 2. Springer, 2007.

Koller and N. Friedman. Probabilistic Graphical Models - Principles and Techniques . MIT Press, 2009.

Meek. Causal Inference and Causal Explanation with Background Knowledge. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence , UAI'95, page 403-410, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603859.

Park and G. Raskutti. Learning quadratic variance function (qvf) dag models via overdispersion scoring (ods). J. Mach. Learn. Res. , 18:224-1, 2017.

Peters and P. Bühlmann. Identifiability of gaussian structural equation models with equal error variances. Biometrika , 101(1):219-228, 2014.

Rebane and J. Pearl. The recovery of causal poly-trees from statistical data. Int. J. Approx. Reason. , 334 2(3):341, 1988. 335

Shimizu, P. O. Hoyer, A. Hyvärinen, A. Kerminen, and M. Jordan. A linear non-gaussian acyclic 336 model for causal discovery. Journal of Machine Learning Research , 7(10), 2006. 337

Spirtes and C. Glymour. An algorithm for fast recovery of sparse causal graphs. Social science 338 computer review , 9(1):62-72, 1991. 339

G. Valiant. A theory of the learnable. In STOC , pages 436-445. ACM, 1984. 340

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes an approach for learning bounded in-degree polytrees that a family of Bayesian networks. More precisely, given the skeleton of the polytree $P$ from which the samples are from, their algorithm learns a $d$-polytree whose distribution is likely to be close to $P$ (with respect to KL divergence) using mutual information tests. Importantly, the algorithm runs in polynomial time for a fixed $d$, whereas the exact learning problem is known to be NP-hard for $d > 1$.\n\nSTRENGTHS:\nTo my knowledge, the theoretical results are novel and show that even though the optimal $d$-polytrees are hard to learn exactly, it can be done approximatively.\n\nWEAKNESSES:\nMy main concern is in the relevance of the article to AI community, i.e., it has nice theoretical results, but their practicality remains unclear to me (see Questions section of this review). I would be happy to increase my score if the authors can offer convincing arguments for this.\n\nI also recommend carefully proofreading the paper to improve its presentation. To mention some of the minor issues:\n\n- 139: \"We denote $\\pi(v)$ to denote\"\n- 143: The definition of deg-l v-structure should probably include the lack of edges between $u_i$ and $u_j$? Of course, that holds implicitly for forests.\n- 143: \"We say that -- is said to be\"\n- 153: Meek [1995] -> [Meek, 1995]\n- 186: has -> have\n\nQUESTIONS:\nWhat kinds of instances would the described algorithms be practical for? In the description of the Algorithm 3, you iterate over all $O(n^d)$ sets of neighbors of size $d$ and compute the estimated mutual information for them. If additionally the number of samples is of order of magnitude $2^d n / \\epsilon$ (line 242), for how large $n$ and $d$ would the described algorithm run in a reasonable time?\n\nIn Lemma 5, you state that the mutual information tests succeed with probability at least $1 - \\delta$. However, for example, line 197 states that \"the algorithm does not make mistakes for orientations\" and line 201 seems to imply that $\\hat{I}$ must always be less than $\\epsilon$. Am I misunderstanding something or shouldn't there be a risk of erroneus orientations?\n\nLIMITATIONS:\nSee Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the number of samples to learn a particular class of distributions: bounded-degree polytrees (Bayesian networks whose skeleton is a forest). Recent work has shown that tree-structured Bayesian networks (1-polytrees) are learnable with finite samples; this work makes progress on the natural generalization to polytrees, showing a positive result when the skeleton is given. The work also provides some conditions under which the skeleton is learnable, and a lower bound for the number of samples required.\n\nSTRENGTHS:\nLearning a distribution approximately from finite samples is one of the most fundamental tasks in learning theory. This study of the finite-sample learnability of polytrees is a very natural step for building our understanding of this problem, particularly in the context of the recent work showing learnability for tree-structured models.\n\nThe main result of Theorem 1 (finite-sample learnability of degree-bounded polytrees given the skeleton) is quite fundamental. The algorithm and proof are generally quite natural, and furthermore they help demonstrate the clean manner in which the mutual information tester machinery of [Bhattacharyya et al., 2021] can be leveraged for such results. While accompanying results in Section 4 (Skeleton assumption) and Section 5 (Lower bounds) are less surprising, their presence adds more completeness to the general picture.\n\nThe paper is generally well-written.\n\nWEAKNESSES:\nMore motivation for studying polytrees might be appreciated by the general NeurIPS community. Regardless, Bayesian networks are well-motivated and polytrees are a natural continuation of the aforementioned recent work.\n\nThe assumption of being given the skeleton is perhaps the most unsatisfying aspect of these results. For context, my understanding is that when learning tree-structured models (as is the focus of the main prior work of [Bhattacharyya et al., 2021]), the entire task is determining the skeleton, as any rooting of the tree is equivalent. In this sense, it is somewhat disappointing that the entire task of the main prior work needs to be given to the polytree learning algorithm. It would be nice to know whether this assumption is inherently required or just an artifact of the current algorithm.\n\nQUESTIONS:\nIs there more discussion that you could provide regarding the necessity of assuming the skeleton is known? Here are some questions that may help orient the discussion (although I do not necessarily expect these to be reasonable to answer in this scope):\n* Generally, is there clear intuition whether a similar result to Theorem 1 should hold without being given the skeleton? On one hand, it seems plausible to imagine that if the skeleton is hard to learn, then perhaps the choice of skeleton is not so important. On the other hand, it seems plausible that it is hard to learn the skeleton and there are many approximately correct skeletons, but orienting an only approximately correct skeleton is hard (maybe this has some connection to the hardness in the unrealizable setting).\n* Is it clear that the Chow-Liu algorithm does not learn an approximately correct skeleton? (Of course, whether having such a skeleton is helpful seems not necessarily obvious.)\n\nMinor remarks:\n* Should it be $\\hat{I}$ in line 145?\n* Line 194 “algorithm 1” should be capitalized/linked.\n* Line 269 “or” -> “of”\n* Generally, it seems like there may be minor errors in comments involving $I, \\hat{I}$, and $C$. For example, on lines 422-423 it says “$\\hat{I}(\\dots) \\le \\varepsilon$. Since $0<C<1$, this implies that $\\hat{I}(\\dots)\\le C \\cdot \\varepsilon$”. Since $C<1$ it is not clear to my why such an implication would hold. A similar remark is made on lines 436-437.\n\nLIMITATIONS:\nThe limitations are addressed fairly in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper introduces an efficient learning algorithm for bounded degree polytrees and establishes finite-sample guarantees. Explicit sample complexity and polynomial time complexity are provided. An information-theoretic lower bound is provided, which shows that the sample complexity of the algorithm is nearly tight.

STRENGTHS:
The paper provides a novel algorithm for learning d-polytrees with general d, extending a previous algorithm for d=1. The theoretical analysis shows that the algorithm is nearly tight in terms of sample complexity. The results do not require distributional assumptions such as strong faithfulness. The ideas and results are clearly presented in the paper.

WEAKNESSES:
The recovery of the true skeleton relies on Assumption 11. It would be nice if some comments on this assumption could be given (e.g. whether it is expected to be tight)

QUESTIONS:
The recovery of the true skeleton relies on Assumption 11. Is this assumption expected to be tight, and what is the obstacle for skeleton recovery in more general scenarios?

In line 233, there seems to be a redundant "then"

LIMITATIONS:
The authors have adequately addressed the limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives an efficient PAC-learning algorithm for learning graphical models called \"bounded polytrees\". These are distributions where 1) the undirected skeleton of the graph is a forest and 2) the in-degree of every node is bounded by some constant $d$. This extends a recent result [1] for directed trees, which is corresponding to the case $d=1$.\n\nIn contrast to [1], the paper gives a learning algorithm assuming that the skeleton is given. To achieve that, the estimator of conditional mutual information from [1] is extensively used. This estimator is used in a sequence of clever greedy-like checks in order to orient as many edges as possible. After orienting the remaining edges, it is shown that the resulting distribution must have small KL divergence to the true distribution.\n\nA sufficient condition is also given, under which the skeleton can be learned for certain distributions by the Chow-Liu algorithm (so it does not have to be given to the algorithm). Finally, a lower bound on sample complexity is proved, roughly matching the upper bound of the algorithm in the case of binary alphabet.\n\n[1] Bhattacharyya, Gayen, Price, Vinodchandran, \"Near-optimal learning of tree-structured distributions by Chow-Liu\", STOC 2021.\n\nSTRENGTHS:\n* The studied problem of efficient learning of graphical models is important and interesting.\n\n* The paper considers distributions with a tree skeleton and arbitrary orientation of edges as opposed to just directed trees. This is a natural and long-studied class of distributions.\n\n* Even given the estimator from [1], the algorithm and proofs are interesting and not trivial.\n\n* Section 3 gives a good outline of the algorithm and its correctness proof and the figures were helpful to me.\n\nWEAKNESSES:\n* The algorithm requires the skeleton as input, which I think is a significant limitation. It is not clear how useful is the sufficient condition proposed by the authors in order to remove this limitation.\n\n* The writing could be clearer. Especially the steps which I assume are more standard/obvious to the authors felt rushed. In my opinion, a few places could be rewritten in order to be clearer and more self-contained.\n\nQUESTIONS:\n* The proof of Theorem 1 in lines 219-227 is very fast. You state the part about estimating the conditional distributions in a conclusory way. I thought the reference for this would be Theorem 1.4 in [1], but you cite two other papers instead. And the sample complexity there seems to have factor $|\\Sigma|^{d+1}$, not $|\\Sigma|^d$. Sorry if I am misunderstanding. \nThe formulas in lines 225-227 are given without any justification and connecting them to the discussion before.\n\n* In the proof of Lemma 7, lines 421-422, I cannot see why Phase 1 guarantees the existence of this vertex. Can you explain? (By the way, the proof says $u\\in S$ and the statement $u\\in S\\cup S'$. Which is it?) (Also I don't understand why it says $\\hat{I}<\\epsilon$ if the algorithm is always checking against $C\\epsilon$.)\n\n* Section 4 also moves fast. In Assumption 11, do you mean that given $P$ and $G^*$, the assumption holds for those $P$ and $G^*$ (and then $G^*$ will be recovered by Chow-Liu run on $P$)? This is not clear from the writing. Also, is $P$ any distribution or is it coming from a tree?\n\n* In section 5, Lemma 14 seems given without proof as a direct consequence of Lemma 13. Shouldn't you also exclude the possibility that there exists a distribution $X\\leftarrow Z\\rightarrow Y$ which is close both to $P_1$ and $P_2$?  \nAlso the argument for Theorem 15 is sketchy. I would appreciate a proof in the appendix.  \n\n* There seem to be some basic properties that you keep applying without ever mentioning them. For example, if $v$ is a node with in-neighbors $u_1,\\ldots,u_k$ then $u_1,\\ldots,u_k$ are independent. And then there is a similar fact with out-neighbors and conditional independence. It would be nice to state those facts at least once in the preliminaries.\n\n* Similarly, it would have helped me if you reminded me from time to time when you are applying formula (1).\n\n* lines 202-204: Am I understanding correctly that the formula you are giving here is the KL divergence between $P=P_{G^*}$ and $P_{\\hat{G}}$? If yes, why not write it explicitly? Why are you using the word \"essentially\" which suggests to me that the formula is not entirely correct? Is there a problem I am not seeing?\n\n\n\nminor:\n* I think sometimes you use $d$ as the true maximum in-degree and sometimes as a variable (e.g., Algorithm 3, proof of Lemma 6). I would avoid this.\n\n* Similarly, the way you use $\\epsilon$ and $\\epsilon'$ is confusing. The value of $\\epsilon$ in Lemmas 7-10 is what you call $\\epsilon'$ in the proof of Theorem 1, correct? If yes, maybe you could give the formula for $\\epsilon'$ before you start analyzing the algorithm.\n\n* I did not understand much from lines 111-116. In the first sentence, do you mean we can always obtain such a graph for any distribution? I guess you can always take the complete graph, but probably that's not what you mean?\n\n* In line 183, can you justify the $O(n^d)$ bound?\n\n* line 192, $d$ should be $d^*$?\n\n* Line 196-197, do you mean not identified in phase 1, or phases 1 and 2?\n\n* Proof of Lemma 21, second and third line after 491, $I(X;Y)$ should be $I(Z;Y)$?\n\n* typos line 59 \"are\", line 94 \"denotes\", line 186 \"has\", line 198 \"in\", double-check the notation in the caption of Figure 3,\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}