Structure learning in polynomial time: Greedy algorithms, Bregman information, and exponential families

Abstract: Greedy algorithms have long been a workhorse for learning graphical models, and more broadly for learning statistical models with sparse structure. In the context of learning directed acyclic graphs, greedy algorithms are popular despite their worst-case exponential runtime. In practice, however, they are very efficient. We provide new insight into this phenomenon by studying a general greedy score-based algorithm for learning DAGs. Unlike edge-greedy algorithms such as the popular GES and hill-climbing algorithms, our approach is vertex-greedy and requires at most a polynomial number of score evaluations. We then show how recent polynomial-time algorithms for learning DAG models are a special case of this algorithm, thereby illustrating how these order-based algorithms can be rigourously interpreted as score-based algorithms. This observation suggests new score functions and optimality conditions based on the duality between Bregman divergences and exponential families, which we explore in detail. Explicit sample and computational complexity bounds are derived. Finally, we provide extensive experiments suggesting that this algorithm indeed optimizes the score in a variety of settings.


Reviews and author responses:

 Thank you for the review and comments. We address the queries below, and apologize in advance for the length of our response, as the questions asked are important and we feel deserving of some careful thought.

**Assumption 4.3:** We are happy to add some intuition behind Assumption 4.3, as follows: Let us first consider the special case of a linear model with independent noise variables. Then it can be shown that Assumption 4.3 implies that $W[j, i]$ is nonzero for any edge $(j, i)$ in the true DAG. This is precisely the traditional causal minimality condition; in other words, for linear models, Assumption 4.3 implies causal minimality. We will be happy to add this result and its (short) proof in the final version of the paper. We are also happy to provide the proof for the reviewer if they would like to see the details.

More generally, Assumption 4.3 is an information inequality that applies when there is a parent $j$ of $i$ that's missing in $Y$. This captures the importance of this edge (j, i), similar to causal minimality as above. In other words, the inequality says that if we miss conditioning on even one parent of $i$, then the conditional Bregman information is higher than if we had conditioned on all the parents of $i$. In other words, the $\phi$-Bregman-information we use captures the "gain in information" that occurs by not conditioning on $X_j$, and all edges from parents should be effective to provide information (recall that conditioning reduces the Bregman-information (L260, App D)).

For concreteness, consider the following simple example: Take 3 variables $X_1, X_2, X_3$ with edges $(X_1, X_2), (X_1, X_3)$. Then, $pa(2) = \\{1\\}$, $A_W(2) = \\{1, 3\\}$. If we pick $i = 2$, $Y = \\{\\}$, then the assumption reads $E[I_{\phi}(X_2)] > E[I_{\phi}(X_2 | X_1)]$. We will see how this inequality captures the fact that edge (1, 2) is needed in the causal DAG. Assume we're working in the traditional setting of the least squares score, corresponding to $\phi(x) = x^2$ so this inequality reads $var(X_2) = E[var(X_2)] > E[var(X_2 | X_1)]$. By the law of total variance, this simplifies to $var(E[X_2 | X_1]) > 0$, which simply says that $X_1$ has a nontrivial influence on $X_2$ in the sense that $E[X_2|X_1]$ is not independent of $X_1$. 
Finally, to compare with faithfulness, add one more edge $(X_3, X_2)$ to the graph. Now Assumption 4.3 simply says all three edges are nontrivial. Faithfulness, however, requires that the two paths $X_1 \to X_2$ and $X_1\to X_3 \to X_2$ do not cancel. Therefore, faithfulness requires “no path-cancellation” in addition to minimality, and is stronger than Assumption 4.3.

We are grateful to the reviewer for bringing to our attention that this assumption may confuse the reader, so for the sake of clarity, we will incorporate the above discussion in the revision of the text. 

**Assumption 4.4:** This assumption can indeed be thought of as a generalization of the equal variance assumption from prior works (see [1, 2, 3]). It is _not_ necessary to assume exact equality, as discussed in Appendix G. In both practice and theory, it suffices for this assumption to hold approximately and not necessarily exactly. This is evident from the inequalities we use in the main proofs in Appendix F and the discussion in Appendix G.

Apart from enabling a theoretical identifiability proof (theorem 4.6) under this assumption, an important contribution of our work is to emphasize that depending on the model and the function $\phi$, the assumption is actually naturally motivated and may be expected to hold approximately. In particular, in L273-283, we illustrate this in the case of a multiplicative SEM models, relevant in growth models stemming from economics and biology, where we choose $\phi(x) = -\log x$. We formalize this in Appendix E, concluding with the proposal of the Itakuro-Saito score which to the best of our knowledge has not been considered before in the literature. The takeaway is that this assumption can indeed be expected to hold in settings which are not equal variance, as was the case with prior works. We leave it for future work to explore this direction for other models and choices of $\phi$ and exploit these observations, with theoretical underpinnings guaranteed by our work.

**Significance:** We highlight two major points: First, an important contribution is the re-interpretation of the several recent algorithms in the literature (for e.g., [2, 3, 4, 5]) as score-based algorithms for different choices of the score-function optimized via a simple greedy scheme (GFBS). In the Bayesian network learning literature, algorithms have broadly been classified into 2 categories: a) constraint-based algorithms that test conditional independence of variables and b) score-based algorithms that try to optimize a score function. Recently there have been a flurry of algorithms that seemingly do something different, in particular some of the works cited above. Bringing them back under the umbrella of score-based learning serves to further our understanding of the power of score-based learning algorithms, as well as bring up several exciting research questions.

The second major contribution is our theoretical guarantees: We show that GFBS with appropriate score functions will provably recover the model (Theorem 4.6). We also derive the sample complexity of GFBS (Theorems 5.4, 5.6). As pointed in L228-232, the Bregman score is directly related to the log-likelihood (whose importance in ML cannot be understated) for exponential family DAGs and considering the well-known approximation capabilities of exponential families, we expect that the Bregman score and our theoretical results apply to a wide variety of settings. 

We remark that these strong theoretical guarantees are backed by extensive experiments that illustrate the power of the GFBS algorithm. The significance is that simple greedy algorithms with the appropriate (and relatively less explored) score-functions can do much better than expected for the important task of causal inference. And by virtue of being simple, they are easily implemented in practice.

Finally, the task of causal inference itself has far reaching implications in machine learning, in particular on the topic of fairness and interpretability that have recently gained tremendous traction in the community. So, research on Bayesian network learning is important progress towards these directions.

**References:**

[1] Peters, J. and B¨uhlmann, P. (2014). Identifiability of Gaussian structural equation models with equal error variances. Biometrika, 101(1):219–228.

[2] W. Chen, M. Drton, and Y. S. Wang. On causal discovery with an equal-variance assumption. Biometrika, 106(4):973–980, 09 2019. ISSN 0006-3444. doi: 10.1093/biomet/asz049. 

[3] M. Gao, Y. Ding, and B. Aragam. A polynomial-time algorithm for learning nonparametric causal graphs. arXiv preprint arXiv:2006.11970, 2020

[4] G. Park and G. Raskutti. Learning quadratic variance function (QVF) dag models via overdispersion scoring (ODS). The Journal ofMachine Learning Research, 18(1):8300–8342, 2017.

[5] Gunwoong Park and Hyewon Park. Identifiability of generalized hypergeometric distribution (ghd) directed acyclic graphical models. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 158–166. PMLR, 2019.

[6] Wang, X., Du, Y., Zhu, S., Ke, L., Chen, Z., Hao, J., & Wang, J. (2021). Ordering-Based Causal Discovery with Reinforcement Learning. arXiv preprint arXiv:2105.06631. 


 We are grateful to the reviewer for their detailed review and thank them for their careful reading of our paper. We appreciate their acknowledgement that our work is “not merely a combination of known techniques” and that “the use of Bregman information as the score function and its analysis is also new”.

Below, we briefly try to address the questions raised. We apologize in advance for the length of our response, as the questions asked are important and we feel deserving of some careful thought.

1. First, we agree that the update step is unfortunately missing, which is precisely $W=W[T→i]$ as the reviewer correctly worked out and this comes right after L4. For the first query, it’s indeed the case that the algorithm as stated has some degeneracy in the minimization step and we thank the reviewer for bringing this to our attention. For _decomposable_ scores, the algorithm is essentially unchanged once we replace this expression by $argmin_{i \not\in T} S_i(v_T)$ as described in L142-146. For _non-decomposable_ scores, the algorithm can be fixed, but this requires some complicated modifications (e.g. finding an optimal source which requires an additional outer loop); so for simplicity, we propose to restrict attention to decomposable scores in the final version. This will simplify the presentation and also align with existing work on greedy optimization that also assumes decomposability. Importantly, please note that the rest of the paper already implicitly assumes the score is decomposable, since least squares scores, Bregman scores are all decomposable scores. In particular, all the main theorems in this paper will be unchanged after this update.

The backward stage acts as a pruning step to erase superfluous edges and is a standard step in DAG learning algorithms (e.g. GES). Under Assumptions 4.3, 4.4, the order doesn’t matter as justified in L679-681 (see the proof of Theorem 4.6 in Appendix D). In the practical setting, this step could be modified in several ways by the algorithm designer. We explore one possibility in L147-148 which also serves to prove our sample complexity upper bound (see definition 5.5). Retrospectively, we think this could have been mentioned earlier. We will clarify and highlight these details in the revision of the text.

2. Apart from [1], other works include [2, 3, 4], and we apologize for this oversight. We remark that the algorithms considered in [3, 4] are indeed special cases of our algorithm, but the scores they use are not necessarily Bregman scores. This highlights the wide applicability of the GFBS algorithm and proposes several interesting future research directions. We will update the text to include this. An important contribution of our research is to unify these different algorithms under a single framework, thereby offering more insight on score-based learning.

3. We are happy to add some intuition behind Assumption 4.3, as follows: Let us first consider the special case of a linear model with independent noise variables. Then it can be shown that Assumption 4.3 implies that $W[j, i]$ is nonzero for any edge $(j, i)$ in the true DAG. This is precisely the traditional causal minimality condition; in other words, for linear models, Assumption 4.3 implies causal minimality. We will be happy to add this result and its (short) proof in the final version of the paper. We are also happy to provide the proof for the reviewer if they would like to see the details.

More generally, Assumption 4.3 is an information inequality that applies when there is a parent $j$ of $i$ that's missing in $Y$. This captures the importance of this edge (j, i), similar to causal minimality as above. In other words, the inequality says that if we miss conditioning on even one parent of $i$, then the conditional Bregman information is higher than if we had conditioned on all the parents of $i$. In other words, the $\phi$-Bregman-information we use captures the "gain in information" that occurs by not conditioning on $X_j$, and all edges from parents should be effective to provide information (recall that conditioning reduces the Bregman-information (L260, App D)).

For concreteness, consider the following simple example: Take 3 variables $X_1, X_2, X_3$ with edges $(X_1, X_2), (X_1, X_3)$. Then, $pa(2) = \\{1\\}$, $A_W(2) = \\{1, 3\\}$. If we pick $i = 2$, $Y = \\{\\}$, then the assumption reads $E[I_{\phi}(X_2)] > E[I_{\phi}(X_2 | X_1)]$. We will see how this inequality captures the fact that edge (1, 2) is needed in the causal DAG. Assume we're working in the traditional setting of the least squares score, corresponding to $\phi(x) = x^2$ so this inequality reads $var(X_2) = E[var(X_2)] > E[var(X_2 | X_1)]$. By the law of total variance, this simplifies to $var(E[X_2 | X_1]) > 0$, which simply says that $X_1$ has a nontrivial influence on $X_2$ in the sense that $E[X_2|X_1]$ is not independent of $X_1$. 

Finally, to compare with faithfulness, add one more edge $(X_3, X_2)$ to the graph. Now Assumption 4.3 simply says all three edges are nontrivial. Faithfulness, however, requires that the two paths $X_1 \to X_2$ and $X_1\to X_3 \to X_2$ do not cancel. Therefore, faithfulness requires “no path-cancellation” in addition to minimality, and is stronger than Assumption 4.3.
We are grateful to the reviewer for bringing to our attention that this assumption may confuse the reader, so for the sake of clarity, we will incorporate the above discussion in the revision of the text. 

4. These assumptions are very standard in the nonparametric statistics literature. It is important to note that _some_ kind of regularity (Holder, Sobolev, Besov, etc.) is necessary for uniform nonparametric estimation to be possible. This is well-documented, see e.g. Ch 1 & 3 of [5]. For example, in Sec 1.7, [5] write “In order to get nontrivial rates of convergence, one has to restrict the class of distributions, e.g., by imposing some smoothness assumptions on the regression function.” 

With this in mind, the Holder class assumption in Assumption 5.2 is standard and the second assumption roughly requires finite fourth moments ($\phi_1=x^2$) or finite $\epsilon$th moment for some $\epsilon>0$ ($\phi_2=\log x$), which is even weaker.

5. We thank the reviewer for their advice on the experiments. Though SHD results are presented in Appendix H.4.2, we are happy to further show the probability of recovery v.s. sample size in the final version, with settings under larger dimensions/number of nodes. In addition, the experiment setups ER/SF-1 and ER/SF-4, where the integer indicates the average degree for each node, are used to compare performances under sparse and dense graphs.

**References:**

[1] W. Chen, M. Drton, and Y. S. Wang. On causal discovery with an equal-variance assumption. Biometrika, 106(4):973–980, 09 2019. ISSN 0006-3444. doi: 10.1093/biomet/asz049. URL https://doi.org/10.1093/biomet/asz049.

[2] M. Gao, Y. Ding, and B. Aragam. A polynomial-time algorithm for learning nonparametric causal graphs. arXiv preprint arXiv:2006.11970, 2020

[3] G. Park and G. Raskutti. Learning quadratic variance function (QVF) dag models via overdispersion scoring (ODS). The Journal ofMachine Learning Research, 18(1):8300–8342, 2017.

[4] Gunwoong Park and Hyewon Park. Identifiability of generalized hypergeometric distribution (ghd) directed acyclic graphical models. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 158–166. PMLR, 2019.

[5] Györfi, László, et al. A distribution-free theory of nonparametric regression. Vol. 1. New York: Springer, 2002.

 We thank the reviewer for their positive comments and for acknowledging the main objectives of our submission: To unify and generalize existing work by introducing the setting of Bregman scores and greedy optimization. We hope this research serves both as a unifying tool for many recent greedy algorithms as well as revive the community’s interest in score-based algorithms and open venues for future work.

 Thank you for the review and for the helpful suggestions. 

> _the paper does not excel in either theoretical (for eg, showing conditions under which the algorithm is indeed optimal)_

We urge the reviewer to take a closer look at our full slate of results (Sections 3-5), especially Section 4.2 (identifiability) and Section 5 (sample complexity). We have included detailed theoretical results showing optimality and correctness (with 8 pages of proofs in the supplement). As another reviewer has pointed out, “The main contribution of this paper is theoretical. The authors provide solid and detailed proofs for their main results as well as experimental illustration”.

Indeed, we **provide explicit conditions under which our algorithm returns the correct structure on both finite and infinite samples** (Theorem 4.6, under Assumptions 4.3, 4.4). As an immediate corollary, we also get a proof of identifiability (Corollary 4.7). We also have sample complexity results (Theorems 5.4, 5.6). In general, our study of the Bregman divergences and the Bregman-score is backed by strong theoretical guarantees that we provide for our algorithm. The experimental results serve to confirm and further the theory.

> _They posit that their approach typically finds the optimal structure (without proof)_

Presumably, the reviewer is referring to our comments between L59-66, especially L65-66. We apologize for any confusion, and thank the reviewer for alerting us to the subtlety of this discussion, which was not adequately addressed in the initial submission. Our submission indeed contains detailed theoretical results on the correctness of GFBS (Theorem 4.6, Corollary 4.7, Theorem 5.4, Theorem 5.6) and explicit assumptions (Assumptions 4.3, 4.4, 5.2). We also note several other theoretical results of independent interest such as Proposition 3.1, Corollary 3.2, Lemmas 4.2, 5.3.

Regarding our comments at L59-66: There is a crucial difference between finding the **correct structure** and finding an **optimal scoring structure**. _These are distinct and need not be the same thing_ (especially on finite samples). See Example 1 in [1] for an explicit example. The comments at L59-66 clarify that in general, we cannot show that GFBS finds a global score minimizer (which is to be expected, since otherwise P=NP). It is an interesting open problem to resolve whether or not GFBS returns a global minimizer of the Bregman information score under some assumptions. We also note that classical score-based algorithms such as GES do not come with global optimization guarantees, and as such there is precedent for this style of result (correctness without global score optimality).

In hindsight, this subtle discussion should have been laid out more clearly early on. Namely, we do provide detailed correctness guarantees, but there is still more to be said regarding global minimization. We will be happy to add this discussion in the final version.

> _Their algorithm (GFBS) seems to perform much better for some scoring functions (log x) than it does for others (x^2) (in that it seems to perform the best for the latter but not for the former) - I'm not sure I understood if there was a difference in the setup of these different bregman information scores._

This is an astute observation, and we can confirm that the experiment setups for different Bregman score functions **are exactly the same**. The difference in performances is simply because the conditional expectation of the multiplicative model ($\phi_2=\log x$) is harder to estimate than the additive model ($\phi_1=x^2$). We will be happy to clarify this in the final version.

> _The evaluation was only performed on relatively small graph sizes (restricted to d = 5). And the experiments were only performed on synthetic data. While testing on synthetic data is reasonable, if the algorithm is primarily a practical one, then describing the performance on larger, real world graphs would have been nice._

For the main experiment on score optimization, due to the expensive computational cost of Gobnilp, which serves as a golden standard, we only present results for $d=5$ (this was mentioned at L342-343). We stress that the comparison to Gobnilp is **necessary** here in order to compare vs the exact global minimizer. Nonetheless, GFBS is polynomial time and is able to handle large $d$. We also provide experiments on graph recovery performance (SHD) in Appendix H.4.2 for moderate dimensions $d=20,30$. Regrettably, we see now that these additional results were not mentioned in the main submission, which we will correct in the final version. We hope the reviewer will go back and review our complete set of additional results in the appendices.

Regarding real world graphs, scale-free networks that we run experiments on are known to provide a reasonable approximation to real world graphs. Since our algorithm captures and generalizes existing works, a more complete evaluation on real data and larger networks can be found in those works. Finally, many more simulations on larger graphs can be found in the supplement.

> _Given that running time/efficiency for these algorithms is important (and is highlighted in the paper), it would have been nice if the experiments also showed the running times._
We thank the reviewer for this suggestion. Here, we append running times of the experiments. We consider two of the more difficult settings described in Appendix H.1: 

- $\phi_1$ function 
- SF-2 graph type
- *LIN* **and** *NGP* model types
- Standard Gaussian noise 
- Fix sample size to be $n=1000$
- Run algorithms with dimension $d=5,10,20,30,40$
- Average running time (in seconds) over 10 replications is reported in the following table. 

 The first 5 columns are for *LIN* model, and the next 5 columns are for *NGP* model. The slowness of *NGP* model is from nonlinear function estimation.

| $d$ | 5    | 10 | 20 | 30 | 40|  5    | 10 | 20 | 30 | 40|
| --- | --- | --- | --- | --- | --- |--- | --- | --- | --- | --- |
| GFBS  |0.045| 0.134| 0.639| 1.288| 2.678|0.554| 4.631| 54.51| 249.8| 811.7|
|NOTEARS| 0.347| 0.439| 1.711| 2.733| 5.042|4.149|7.078| 18.05| 41.82| 48.16|
| GES| 0.344| 0.359| 0.407| 0.463| 0.527|0.316|0.384| 0.413| 0.418| 0.481|

> _for eg line 63 "In general, learning Markov random fields is NP-hard [41], even for trees [13]" - I believe learning undirected trees optimally is possible in polynomial queries using Chow-Liu_

The reviewer is correct here, and we apologize for the oversight. The reference [13] is for learning _directed_ polytrees, which is NP-hard, but for _undirected_ trees the Chow-Liu algorithm is indeed optimal. We thank the reviewer for pointing this out and we will fix this in the final version.

**References:**

[1] Loh, Po-Ling, and Peter Bühlmann. "High-dimensional learning of linear causal networks via inverse covariance estimation." The Journal of Machine Learning Research 15.1 (2014): 3065-3105.


The authors propose a greedy algorithm for learning graphical model structure, based on adding nodes in a a greedy forward pass, and then removing edges in a backward pass. The nature of the algorithm ensures that it requires at most polynomial score evaluations. They compare their approach to existing approaches experimentally. They posit that their approach typically finds the optimal structure (without proof)
  The paper is clear, and well written. But overall, I felt that this does not rise to a level of an accept for a few reasons.
1) The algorithm itself is nice / straightforward. However, the paper does not excel in either theoretical (for eg, showing conditions under which the algorithm is indeed optimal) or experimental analysis.

The experimental concerns come down to
1) Their algorithm (GFBS) seems to perform much better for some scoring functions ($log(x)$) than it does for others ($x^2$) (in that it seems to perform the best for the latter but not for the former) - I'm not sure I understood if there was a difference in the setup of these different bregman information scores.
2) The evaluation was only performed on relatively small graph sizes (restricted to d = 5). And the experiments were only performed on synthetic data. While testing on synthetic data is reasonable, if the algorithm is primarly a practical one, then describing the performance on larger, real world graphs would have been nice.
3) Given that running time/efficiency for these algorithms is important (and is highlighted in the paper), it would have been nice if the experiments also showed the running times.

The writing was on the whole pretty good, but there were some nits (for eg line 63 "In general, learning Markov random fields is NP-hard [41], even for trees [13]" - I believe learning undirected trees optimally is possible in polynomial queries using Chow-Liu) 

 Optimal (or near optimal) structure learning has a variety of impact for learning underlying causal mechanisms, and going beyond "curve fitting".

This paper studies the problem of learning graphical models defined on DAGs. The authors propose a general vertex-greedy score-based algorithm that requires polynomially many score evaluations.  This paper studies sample and evaluation complexity for learning DAGs using greedy score-based algorithms. The greedy algorithm presented in this paper is very general and runs with polynomial computational complexity. Previous algorithms for learning DAGs are usually edge-greedy by adding or removing edges greedily, while the algorithm in this paper is vertex-greedy which adds vertices in a topological order. This algorithm in fact generalizes known algorithms in the literature which are order-based, under a suitable choice of score functions. In this sense, the greedy algorithm proposed in this paper is universal and unify a lot of previous works. 

The authors also introduce new score functions based on the Bregman information, and provide theoretical bounds on sample and computational complexity. Also, the authors give evidence that the algorithm in fact optimizes the score function globally in a lot of cases, while it is well-known that the optimization problem is hard in general. This paper is nicely written. The main contribution of this paper is theoretical. The authors provide solid and detailed proofs for their main results as well as experimental illustration.  This is a nice paper. As a very minor suggestion some more discussions and comparisons for other order-based algorithms would be better.


The paper studies the problem of learning the structure of Bayesian networks (DAGs) from fully observational data. The main contributions are:

1. Authors propose a simple greedy algorithm that consists of two stages. The first stage aims to estimate the topological ordering of the DAG by greedily appending the node that minimizes a given score function. Thus, this phase is vertex-based and performs $O(d^2)$ score computations. The second stage consists of a pruning step, where edges, whose presence do not influence much the score of the structure, are deleted.

2. Authors claim that *several* existing order-based algorithms are special cases of the algorithm described above. Thus, it is claimed that those methods are equivalently score-based methods.

3. Authors propose the use of Bregman information as the score function for their algorithm. Using properties of this scoring function, authors derive statistical and computational guarantees for their proposed algorithm.

4. Authors claim that their method recovers the ground-truth network by globally optimizing the score function, and provide experiments to support this argument.  Here I present my review along the axes from the NeurIPS reviewing guidelines. See my questions/comments at the end.

**Originality**

- Are the tasks or methods new? 
  - Yes. The approach of learning the BN structure in a score-based fashion is not new but the presented algorithm is. Also, while the proposed method has some similarities to existing greedy algorithms, authors show in Appendix A that their method is indeed different to edge-based greedy algorithms such as GES.
  - The use of Bregman information as the score function and its analysis is also new to the best of my knowledge.

- Is the work a novel combination of well-known techniques?
  - The work is motivated by known approaches for learning BN structures but is not merely a combination of known techniques.

- Is it clear how this work differs from previous contributions? 
  - Yes. Authors clearly state that their method is score-based, greedy, and how it differs from known greedy algorithms.

- Is related work adequately cited?
  - Yes, given the vast literature in BN structure learning. I think authors do a proper job in citing the most closest existing results.


**Quality** 

- Is the submission technically sound? 
  - Yes.

- Are claims well supported (e.g., by theoretical analysis or experimental results)? 
  - Yes, proofs are provided in the Appendix. Empirical evidence is also provided in Section 6.

- Are the methods used appropriate? 
  - Yes.

- Is this a complete piece of work or work in progress? 
  - This is a complete work.

- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
  - More or less. There are a few statements that were confusing to me, see my comments below.


**Clarity** 

- Is the submission clearly written? 
  - Mostly yes. It can be improved, see my comments below.

- Is it well organized? 
  - Yes. I was able to follow the main results without issues.

- Does it adequately inform the reader? 
  - Yes. E.g., experimental details are given in Appendix H.


**Significance** 

- Are the results important? 
  - Yes. Learning the structure of BNs is long-standing problem in machine learning.

- Are others (researchers or practitioners) likely to use the ideas or build on them? 
  - Perhaps. It might be interesting to see the performance of this method in real-world data.

- Does the submission address a difficult task in a better way than previous work? 
  - Somewhat unclear. The experiments are mostly related to the scores obtained by their method and others. While that is relevant, I believe that the main experiments should be how well the method recovers the true DAG. Figures 4, 5, and 6 in the Appendix attempt to show this but the experimental setting is a bit unconvincing. See below. 

- Does it advance the state of the art in a demonstrable way? 
  - Unclear too. It seems that the method is competitive but more experiments are needed.

- Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
  - No.

# Questions / Comments

1. I spent some time figuring out Algorithm 1, to me, it is confusing. If this is the case, it is disappointing that the only algorithm in the paper is confusing despite being simple. At the first iteration of the forward phase, we have $T = []$ and so we are supposed to compute $\arg\min_{i \in V} S(W[T \to i])$. However, what is $W[T \to i]$ when $T = []$? If I follow the notation, it says that no edges pointing to $i$ are added  but then for all $i \in V$ we would have the same score $S(\emptyset)$. Thus, my first confusion is what happens at the first iteration of the algorithm. Second, if I read correctly, $W$ is not being updated during the forward phase. Then, what edges are we deleting in the backward phase? My guess is that once $i$ is chosen, we do $W = W[T \to i]$, but again I am just guessing. Third, I believe no comments are given regarding the order for deleting edges. Any iteration order will end up with the same final $W$? If so, why?

2. In the abstract and introduction, it is claimed that *several* existing order-based methods are special cases of the proposed greedy algorithm for some score function. However, I think this is formally shown *only* for the method by Chen et al. [5]. If this is the case, then the claim for *several* order-based methods is an overstatement and weakens the contributions of the work.

3. In line 238, it is stated that Assumption 4.3. will "hold in most settings" but no formal or empirical support is given. I actually find the assumption somewhat obscure, and will appreciate if more intuition is given. Also, can authors briefly comment on the pros/cons of this assumption to other known assumptions such as faithfulness? 

4. I also find Assumption 5.2. somewhat restrictive. Can authors comment on this?

5. I find the experiments somewhat inconclusive regarding the recovery of the true DAG. 
I believe having the plot showing the probability of exact recovery vs number of samples for the proposed method would strengthen the manuscript, where Figure 2 only looks for the ordering and not the whole graph. In Figures 4, 5, and 6, it seems that the proposed method can outperform other existing methods. However, to have a better sense of this, the setting with larger number of nodes is needed. I think there is also an implicit sparsity assumption in the experimental setting. Have authors evaluated the performance in dense DAGs?

 Yes.

This paper proposes a general unifying vertex-greedy algorithm (BFGS) for solving the score-based structure learning problem over DAGs. The paper focuses on a family of scoring functions derived from Bregman information. The authors prove that under certain assumptions (4.3: Lack of superfluous edges, 4.4: Equal Bregman-information upon conditioning), the underlying DAG is identifiable using only observed data, and the proposed algorithm can identify it. The authors also give sample complexity guarantees under additional assumptions on the conditional distributions of the variables (Holder continuity and finite second moments).   Originality: The proposed algorithm seems to be a generalization of the algorithm given by Chen et. al.[5]. The arguments and assumptions generalize the idea by replacing variance with Bregman information. This allows the authors to subsume some of the new order-based methods under the general vertex-greedy algorithm. 

Quality: The work seems to be technically sound but the motivation behind assumption 4.3 and assumption 4.4 is not completely clear to me. It would be great if the authors could provide some justification for making the assumptions 4.3 and 4.4 for general functions $\phi$.

Clarity: The paper is easy to understand and provides all the necessary technical details.

Significance: While the work finds a unifying theme for some of the existing order-based approaches, the significance of the observation is not clear to me. I would like to know a bit more about the significance of this result for future work. The authors also mention the relation between Bregman divergence and exponential families in the paper but don't explore its significance later. It would be great if they further explore this relation.  The authors have adequately addressed the negative societal impact of the paper.

Meta-review: This paper studies the problem of learning directed acyclic graphs via a greedy score-based algorithm. Specifically their algorithm works in two stages, first by estimating the topological ordering of the nodes, and then by pruning edges that do not influence the score. Compared to earlier works, the main innovation is in using the Bregman information as the score function. This allows them to show their algorithm recovers the ground truth network under various assumptions. This is a nice contribution, but the main point of contention among the reviewers was whether these assumptions were natural and/or properly justified. In their response, the authors related it to causal minimality, however the reviewers felt that it seemed to be stronger. As a suggestion for the authors, it would be better if Assumptions 4.3 and 4.4 were discussed in more depth and made more of a focal point of the paper. 

Decision: accept