Efficient Bayesian network structure learning via local Markov boundary search

Abstract: We analyze the complexity of learning directed acyclic graphical models from observational data in general settings without specific distributional assumptions. Our approach is information-theoretic and uses a local Markov boundary search procedure in order to recursively construct ancestral sets in the underlying graphical model. Perhaps surprisingly, we show that for certain graph ensembles, a simple forward greedy search algorithm (i.e. without a backward pruning phase) suffices to learn the Markov boundary of each node. This substantially improves the sample complexity, which we show is at most polynomial in the number of nodes. This is then applied to learn the entire graph under a novel identifiability condition that generalizes existing conditions from the literature. As a matter of independent interest, we establish finite-sample guarantees for the problem of recovering Markov boundaries from data. Moreover, we apply our results to the special case of polytrees, for which the assumptions simplify, and provide explicit conditions under which polytrees are identifiable and learnable in polynomial time. We further illustrate the performance of the algorithm, which is easy to implement, in a simulation study. Our approach is general, works for discrete or continuous distributions without distributional assumptions, and as such sheds light on the minimal assumptions required to efficiently learn the structure of directed graphical models from data.


Reviews and author responses:

This paper considers learning DAGs from iid sampled data without specific distributional assumptions. The sample complexity is derived and is shown to be polynomial in the number of nodes. One advantage is that the faithfulness condition that is often assumed in this line of work is no longer required.   Strengths: 

+ Graphical model selection is an important problem in statistical learning. Deriving tight sample complexity bounds is of paramount interest in understanding the fundamental limits of this task.

+ The algorithm seems natural and the analysis seems correct from what I checked. 

Weaknesses:

- The assumptions appear to be too strong. In particular, Condition 2 requires that H(X_k | pa(k)) = h^* does not depend on k. The authors claim that their method works for models without specific distributional assumptions. However, this precise entropy condition violates the claim. This condition is just too strong for any real-world graphical model.

- The sample complexity bound seems to be too weak. The dependence on d (as a quadratic term) is too strong and thus, the algorithm and results do not work well in the high-dimensional regime, a basic requirement. 

- This would have been mitigated if the authors attempted to derive a lower bound in the absence of the faithfulness condition as assumed the PC algorithm. However, the fact that there's no lower bound in the paper dilutes the overall contribution. 

- The numerical experiments are weak. We cannot observe the dependence of the sample complexity on d^2. The improvement in performance compared to PC is not evident. 

- As a suggestion to the authors for a future submission, the authors can consider a particular example that corroborates their claim that the algorithm herein works without the faithfulness assumption by running experiments such that PC fails but the algorithm herein works. This would be really illustrative and useful for the community, and a convincing empirical demonstration of the utility of the work.  None. Theory paper. 

The paper develops a new algorithm for structural learning of directed acyclic graph with specific reference to bayesian networks.
A theoretical argument is developed from which a structural learning algorithms is presented. 
The algorithm consists of two main parts, one which put the learning problem in a general framework and the scond algorithm which can eventually be replaced by other state of the art algorithms.
Results of numeirical experiments witness that the proposed structural learning algorithm, which works for discrete or continuous distributions without any distributional assumptions, performs the same as state of the art algorithms in the case where the number of learning observations increases.  Originality: 
The paper is original because it proposes a new algorithm to learn the structure of bayesian networks from available data. The originality stands by the fact that the paper is based on ancestral formulation of the learning prolem and uses a greedy strategy to achieve optimality.

Quality: 
The paper is well structured and written. Terms are properly introduced and defined before using them.
THe paper reads very well and the narration is sensible while each piece of information isappropriately presented and discussed.
I did not check the proof of theorems but according to theoremes in the paper, the author/s convinced me that their algorithm works.

Clarity:
The paper is clear, in other terms it is is clear the theoretical problem which is tackled and it is also very clear how the problem has been solved.

Significance: 
The paper is extremely significant to the community of machin learning with specific reference to bayesian networks and causal networks. I personally think that the results preseted in the paper will be very useful for other scholars in machine learning and artificial intelligence. I was not able to find any discussion about limitations of the proposed approach. 
The paper devotes some space to potential extensions and future works but no mentions to critics of the proposed approach.
The impact to the reseach community could be good but this strongly depends on the capacity to show what can concretely be obtained in practical applications.

 I think that arguments from sG1J and g49G are relevant and thus I thank them to help me better evaluate this submission.

 At the request from the AC, we’d like to add some additional comments on the $d^2$ dependence and comparison with some existing works which provide sample complexity results [5, 16, 19, 42]. 

**Regarding the $d^2$ dependence:** It is important to note that even in _parametric_ settings, existing work has a $d^2$ dependence, see e.g. [5]. For nonparametric settings ( such as [16]), the sample complexity is exponential; see below for more on this.

**Comparison with existing works:** Most previous work, including [5,19,42] only consider linear and/or parametric models. The only paper that provides nonparametric guarantees (without assuming faithfulness) is [16]. Moreover, in [16] the authors only provide guarantees for recovery of the topological ordering, and **do not consider the harder problem of exact graph recovery**. Furthermore, they do not include the reduction to Markov boundary search that is crucial to our submission, and they of course make a much stronger identifiability assumption (similar to Condition 2).

This makes the results somewhat difficult to compare, since in our submission we consider exact graph recovery (i.e. including parent selection via Markov boundary search), which is tied to learning the topological sort. We also assume $M\le \log d$, while [16] does not (this is again because they do not consider Markov boundaries).

Nonetheless, we can still make the following comparison: The sample complexity in [16] is exponential in $d$: $(dr/\epsilon)^{1+d/2}$ (ignoring other parameters). This exponential dependence in [16] is not surprising since they are in a nonparametric setting, and make no assumptions on $M$. In our submission, assuming $M\le \log d$, the sample complexity scales as $d^2r\log^3 d / \epsilon$ (again, ignoring other parameters). 

**Regarding the sample complexity lower bound:** We are not aware of any lower bounds in the nonparametric setting, nor of any experiments along these lines. For the parametric setting, lower bounds are known [*], and existing works [e.g. 5,19,42] do not match these lower bounds. 

With this in mind, we'd like to reiterate that one of the main thrusts of our work is to make progress on nonparametric problems, since it appears that very little is known. 

[*] Ghoshal, Asish, and Jean Honorio. "Information-theoretic limits of Bayesian network structure learning." Artificial Intelligence and Statistics. PMLR, 2017.


 I appreciate the authors' responses. Thanks. I also read the other reviews and responses.

I believe the example where constraint-based algorithms will definitely fail is very helpful to show how the proposed assumptions lead to different results than the classic approach.

While I won't lower my score,  I also believe the concerns of Reviewer sG1J are all valid.

 Dear authors,

I have already stated my points and am willing to up my score to 5; please do not force the issue any more. I do not think it's necessary for me to spell out specific references. I'm referring to the general area of "graphical model selection" and what are the typical results one can find in typical papers from the ones I'm familiar with. 
To wit, the novelty of papers is a subjective matter. I have already mentioned that this paper suffers from some deficiencies, which are also agreed upon by other reviewers. The question then is whether weighing the strengths and weaknesses of this paper puts it above or below the bar. I have stated where I think it lies and I leave it to the AC to decide; I think I'm being very fair here. Thank you. This will be my final message on this paper. 
Reviewer sG1J

 Thank you for your response and your time! We are happy to discuss further on this point -- since you say you are "just comparing other NeurIPS papers on graphical model selection", would you kindly provide specific references to these other papers on graphical model selection that provide **tight sample complexity bounds** for **directed acyclic graph** selection? 

It is important to distinguish learning _undirected_ graphs from _directed_ graphs, which are distinct problems with a distinct literature. 

 Dear authors,

Thanks for acknowledging that the presentation of the conditions for successful structure recovery can be improved. I think that is a deficiency in this paper. 

Furthermore, the fact that the proved sufficient sample complexity depends on $d^2$ is probably not tight. For these lines of work (graphical model selection), it seems like a necessary condition is necessary to make the work "complete". The way the result is presented and the lack of convincing experiments to support necessity the $d^2$ in the sample complexity is also a weakness in the paper. This reviewer is not demanding -- I'm just comparing other NeurIPS papers on graphical model selection to the present submission. If the derived lower bound is not tight, at least the authors can discuss where the looseness stems from or support the sufficiency condition by thoughtful numerical experiments. However, here there is no lower bound. 

The best I can recommend for this paper is a borderline. I don't know what number that corresponds to and I don't know how to change my numerical recommendation, but I leave it to the AC to decide. 

 Thank you for your quick response, and we are happy to hear you are upping your score in light of our response. 

**Regarding the presentation of the conditions:** You raise an excellent point, and we see how the current presentation can be confusing. We are happy to revise this to make things clearer; perhaps by making this a numbered equation instead of a block environment that stands out. We are more than happy to hear alternative suggestions on how best to present this.

We are also pleased to have clarified the point on nonparametric sample complexity. If there are any other points you'd like to discuss further, please let us know!

 Thanks to the authors for a detailed response. 

Sample complexity: 

I am happy that the authors clarified that the learning is done on non-parametric distributions and so the sample complexity may not be ideal. This stems from the poor concentration of the empirical entropy in Prop. 4.1. 

Strong assumptions: 

I had an objection to the assumption that all the entropies are the same as stated in Condition 2. Thanks to the authors' reply, I see that this is not needed. However, why state Condition 2 so prominently? The minimal set of conditions for the alg to work should be stated, not some restrictive version. 

I'm happy to up my score to 5. 



 We thank the reviewer for the constructive comments. Please find our response below:

**Strong assumption**: It is **not** necessary to assume exact equality. Please see Condition 1, L158-164, Remark 1, Section 3.2, and Appendix B.3 for extensive discussion on this point. Condition 2 (exact equality) is simply a sufficient condition that has been included for completeness and to help situate our work within the existing literature, which makes similar assumptions to Condition 2. Indeed, the exact equality condition has been well-received in the literature [1,2] and can easily be relaxed to inequality as in previous work (see, e.g. Theorem B.5). Beyond Condition 1, which only requires that entropy is well-defined, we make **no distributional assumptions** whatsoever.

**Sample complexity**: Since our setting is nonparametric, it is expected that the curse of dimensionality will preclude the high-dimensional regime $d\gg n$. In fact, polynomial and even exponential in $d$ (e.g. measurement error models) is quite common in nonparametric estimation. Although lower bounds are certainly interesting, our current work is focused on the highly nontrivial problem of efficient algorithms and upper bounds. Nonetheless, we will point out that the $2^M$ dependence is genuine and can be established by consideration of lower bounds. We believe this is appropriate to leave for future work.

**Unfaithful example**: We are grateful for this suggestion from the reviewer, and are happy to share a simple unfaithful example. In fact, such examples are trivial to construct and we regret not including this in the original submission. 

Consider a notorious three-node DAG: $Z\to Y$, $Z\to X\to Y$, where $Z$ is a common cause of $X$ and $Y$, and the effect $Z\to Y$ is cancelled by the path $Z\to X\to Y$. Then independence between $Z$ and $Y$ does not imply the d-separation between them. Now add one more node $W$ with an edge $W\to X$. This Bayesian Network is still unfaithful due to the independence between $Z$ and $Y$, but can easily be made to satisfy our identifiability condition. To see this, in the first layer, $W$ is used to mask $(X,Y)$ (note that $Z$ is unable to, by independence). Then in the second layer, $X$ will mask $Y$. We have run experiments to show PC/GES will have SHD around 3 (i.e. very bad for this small model) but our algorithm will go to zero as expected. 

Furthermore, this simple example can be embedded into any DAG satisfying our identifiability condition by treating some sink node as $W$. And we may conjecture that, when unfaithfulness happens in a form of path-cancellation, if we have some other nodes within the “ancestral” layer, it is able to help identify the DAG as long as the  “uncertainty” relation is satisfied (C1).

**Reference**:

[1] Chen et al. On causal discovery with an equal-variance assumption, 2018.

[2] Gao et al.   A  polynomial-time  algorithm  for  learning  nonparametric  causal graphs, 2020.


 We thank the reviewer for the positive comments, particularly that our approach is “extremely significant” and “very useful for other scholars”. We hope this will be the case as well!

 We thank the reviewer for the helpful comments. Please find our response to the detailed questions below:

**Comparison with existing work / originality**: 
As the reviewer points out, our algorithm is superficially similar to previous ordering-based algorithms, however, we’d like to stress that our algorithm is new: The testing and masking (T&M) step has not appeared previously, and is crucial to generalizing previous algorithms. This is an important insight, and also significantly generalizes existing identifiability results. Specifically, this new technique helps generalize Conditions 2 and 5 (which have appeared previously in the literature) such that our main condition (C1) can cover existing identifiability results (Theorem B.6 and Example 1). This also helps understand the implications of (C1) (see below). 

The relaxation from variance to entropy is also significant: Instead of requiring finite second moments, entropy only requires _some_ (possibly fractional) moment to exist, i.e. $\mathbb{E} X^\epsilon < \infty$ for some $\epsilon>0$.

**Implications of (C1)**: The basic intuition of Condition 2 or 5 is that the entropy (uncertainty) in the entire system increases as time evolves. This can be justified intuitively via physical principles (e.g. thermodynamics requires that the entropy of a system is increasing), but also has important mathematical implications. Compared to previous work, instead of requiring every ancestor in the previous layers to have smaller entropy, just one ancestor in each preceding layer is enough for identifiability, which also sheds light on when (C1) can be violated. Our motivation for this assumption is to understand the necessary conditions for recovery without faithfulness; our results show that previous works (which have been published in NeurIPS in recent years) make significantly stronger assumptions than necessary. Finally, see below for an example where (C1) holds but faithfulness fails to hold.

**Experiments**: We stress that the current experiments are used for simple illustration thus not well-optimized compared to existing algorithms or fully exploited the main condition either (see L306-308). We have made no attempts whatsoever to optimize the algorithm or entropy estimation, tune parameters, or devise clever models. By contrast, the PC and GES implementations are highly optimized to be fast and stable. We speculate that the comparatively poor performance comes from the hardness of entropy estimation and the fact that our synthetic models are all in fact faithful. As another reviewer has pointed out, a more convincing demonstrating of our algorithm would have been on **unfaithful** models, which we are happy to include in the final version (see below for an example). On such models (which are extremely simple to derive, see below), both PC and GES will fail while our algorithm succeeds. Furthermore, these simple examples can be embedded into any model. In hindsight, we regret not illustrating this more carefully in the original submission, and we are happy to talk more about this in detail during the response period. 

**Example**: We conclude with a simple example where PC/GES fail but our algorithm succeeds (both in theory and in practice). Consider a notorious three-node DAG: $Z\to Y$, $Z\to X\to Y$, where $Z$ is a common cause of $X$ and $Y$, and the effect $Z\to Y$ is cancelled by the path $Z\to X\to Y$. Then independence between $Z$ and $Y$ does not imply the d-separation between them. Now add one more node $W$ with an edge $W\to X$. This Bayesian network is still unfaithful due to the independence between $Z$ and $Y$, but can easily be made to satisfy our identifiability condition. To see this, in the first layer, $W$ is used to mask $(X,Y)$ (note that $Z$ is unable to, by independence). Then in the second layer, $X$ will mask $Y$. We have run experiments to show PC/GES will have SHD around 3 (i.e. very bad for this small model) but our algorithm will go to zero as expected. 

Finally, we appreciate the helpful  advice on the writing.


 We thank the reviewer for the positive comments and encouraging remarks. 

We definitely agree that generalizing Condition 3 (PPS condition) is an important direction for future work. On this note, we’d like to emphasize our results in Sections 5.3 (learning polytrees) and Appendix B.4 (PPS condition for non-polytrees), where we construct explicit examples where this condition holds. Unlike undirected graphs where the Chow-Liu algorithm is optimal, learning polytrees is still very much an unresolved problem on which we have made some progress.


In this paper, the authors look at the problem of learning the structure of a Bayesian network from samples. In this line of work, equal variance has been a standard assumption, see for example the work by Ghoshal and Honorio. In this paper, the authors propose an entropic assumption, which they show is more general than the equal variance assumption, in recovering the true structure. In their algorithm, they learn the Bayes net layer after layer. In discovering each layer, they crucially use their entropic assumption to mask out vertices that do not belong to the current layer. Their technique is applicable also in continuous settings, using the differential entropy in place of Shannon entropy. In fact, that is how they claim to generalize the equal variance assumption. Therefore, at a high level, the authors claim that their entropic condition is much more fundamental. The authors also give a formal guarantee of the Markov boundary search procedure used to detect the parents of a given node among its Markov boundary, using a PPS condition. However, I did not find the later condition very realistic in practice.  Finally, the authors also give finite sample upper bounds for recovering the true structure from samples based on certain quantitative (stronger) versions of their assumptions. They also experiment against the current best algorithms and establish that their estimator performs as well and sometimes better than the current estimators. I think this is a good paper and vote for acceptance. Yes

In this work, the authors propose a novel identifiability condition/assumption for learning Bayesian network (BN) structures. They then describe an approach for BN structure learning (BNSL) which first identifies nodes in each layer of the DAG structure of the BN; a second algorithm is then used to identify the parents of each node by identifying its Markov blanket among its candidate ancestors (namely, all nodes appearing in an earlier layer). The identifiability condition is then used to show that the algorithms identify a minimal I-map. Detailed complexity analysis shows that the time and sample complexity are exponential in the size of the largest Markov blanket in the DAG, rather than the number of nodes.  --- Originality

The main novelty of the proposed work is in the new identifiability condition/assumption. The algorithms themselves are quite similar (or exactly the same) as well-known algorithms in the field. The identifiability assumption itself seems reasonable enough, though more implications of this assumption could be considered. Still, it leads to meaningful proofs on algorithmic and sample complexity.

--- Quality

I am not an expert in this area, and I have not thoroughly evaluated the proofs in the appendix. Nevertheless, the intuitions given in the main text seem plausible, and the overall technical quality of the theoretical work appears sound. I read through the python code; it hews closely to the same notation and structure as the algorithms as well as follows standard coding styles.
As the authors acknowledge, the empirical experiments are very minimal and do little more than show a proof of concept; still, though, the results for the proposed approach are quite bad for modest sample sizes and numbers of variables.

--- Clarity

The paper is well-written. Various quantities and concepts important in the text are succinctly described. As hinted above described in “Limitations” below, though, the implications of the new identifiability assumption, particularly (C1), and violations thereof, are not clear to me.

--- Significance

BNSL is a widely-studied problem that has also been applied in various domains. Especially in domains such as biomedical where accurate modeling and limited samples sizes are common, the proposed approach provides a practical way to alleviate uncertainty due to the choice of algorithm. --- Originality

Algorithm 1 is very similar in nature to searching for topological orders/permutations (e.g., [Teyssier and Koller, UAI 2005], but many others as well), while Algorithm 2 is exactly (the forward pass of) IAMB. The new identifiability assumption ensures the greedy algorithms find an optimal solution, but the algorithms themselves are well-known.

--- Quality

The experimental results in Figure 2 are rather disappointing. The main theoretical contributions are polynomial guarantees on algorithmic and sample complexity. However, the size of the networks is relatively small (only up to 50 nodes) while the number of samples is modestly large (at least 1000 samples). Further, the performance of the proposed approach is rather bad compared to simple baseline algorithms in several settings. I appreciate that the theoretical work is the main contribution, but the experiments do not do a good job of highlighting any strengths of this approach.

--- Clarity

As a non-expert, it is difficult for me to understand the implications of (C1). It is unclear to me in what kinds of situations such an assumption may not hold. The authors mention that faithfulness is not important for this approach to work, but it is unclear to me what happens when this condition does not hold. For example, can the algorithm detect such cases? possibly with access to an oracle for the independence tests?

--- Significance

The limited and poor experimental results, combined with the lack of clarity on one of the main assumptions in the work, could limit its impact.

--- Minor comments, typos, etc.

The references are not consistently formatted

“d_6” -> “h_6” (Section 3.2)

“\hat{h}_{jk}” is not defined in Algorithm 1, though is presumably the respective conditional entropy

--- Societal impact

No obvious, direct societal impact.


Meta-review: The final reviewer scores are borderline, with the merits being valuable contributions to the ongoing literature on Bayesian network learning (e.g., generalising the "equal variance" condition, which was a recent success case), but also some concerns.  After carefully discussing and considering these concerns, my overall recommendation is "weak accept".  The reasons are as follows:
- One concern was that Condition 2 is too strong.  However, this was largely a misunderstanding, and the authors did indeed already point out in the paper that the condition is *sufficient* for Condition 1, but that the latter is more general.  Addressing this potential confusion should only be a minor change, e.g., Condition 2 could be labeled with (Sufficient Condition for Condition 1), or changed to regular text (instead of bold-headed environment), etc.
- Another concern was that the d^2 dependence seems high and there are no lower bounds to compare against. However, this dependence appears even in simpler settings (e.g., assuming linearity), and lower bounds are largely open, so insisting on these may be asking too much. Overall, this setup appears to be much more of a work in progress compared to learning undirected Markov Random Fields, so it is natural that more gaps still remain.  Having said this, the authors should carefully compare to the most related works (as they partly did in the discussion) in the revised paper, as currently the related work seems to be written without enough emphasis on the relevant sample complexities and assumptions.  Please also highlight the relevant limitations/gaps/open problems carefully.

Beyond these specific points, I ask the authors to carefully consider all reviewer comments when editing the paper.

Decision: accept