Title: ProDAG: Projected Variational Inference for Directed Acyclic Graphs
Venue: NeurIPS 2025
Accepted: yes
Keywords: structure learning

directed acyclic graph

variational inference
Abstract: Directed acyclic graph (DAG) learning is a central task in structure discovery and causal inference. Although the field has witnessed remarkable advances over the past few years, it remains statistically and computationally challenging to learn a single (point estimate) DAG from data, let alone provide uncertainty quantification. We address the difficult task of quantifying graph uncertainty by developing a Bayesian variational inference framework based on novel, provably valid distributions that have support directly on the space of sparse DAGs. These distributions, which we use to define our prior and variational posterior, are induced by a projection operation that maps an arbitrary continuous distribution onto the space of sparse weighted acyclic adjacency matrices. While this projection is combinatorial, it can be solved efficiently using recent continuous reformulations of acyclicity constraints. We empirically demonstrate that our method, ProDAG, can outperform state-of-the-art alternatives in both accuracy and uncertainty quantification.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel Bayesian causal discovery method ProDAG for learning directed acyclic graphs (DAG) with uncertainty quantification. The prior on the adjacency matrix $W$ of the DAGs is induced by projecting $\\tilde W\\in\\mathbb R^{p\\times p}, \\tilde W\\sim\\mathbb P$ onto its closest neighbor in the set of adjacency matrices. The author showed that the projection is unique and thus the prior is well-defined. Furthermore, the author introduced a variational inference method to learn the underlying DAG based on this prior. Experiments have shown that ProDAG performs competitively against the state-of-the-art (Bayesian) DAG learning methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed method is novel and shows good performance for various noise distributions and graph structures.\n2. The prior distribution is well-motivated and justified.\n3. The projection procedure in the prior evaluation is efficient. Moreover, the author provides explicit gradient estimation for this projection.\n\n**Weaknesses**\n1. Identifiability. The author did not discuss what kind of DAGs the proposed model can identify. Since the VI procedure requires knowledge of the likelihood, it seems that the noise distribution has to be known to guarantee identifiability, which is a limitation that needs to be declared. \n2. Some experiments are not sufficient. For example, the experiment for nonlinear data is limited to $d=10$, and those for non-Gaussian noise distributions are limited to $d=20$. Experiments on higher dimensions, e.g., $d=100$ would be helpful.\n3. Time comparison with other Bayesian methods is not provided.\n\nI will be happy to increase the score if my concerns can be addressed.\n\nQUESTIONS:\n1. Do you assume knowledge of the true noise distribution for experiments on non-Gaussian data?\n2. Is ProDAG robust if the log-likelihood function is misspecified? For example, using a Gaussian likelihood for data generated by non-Gaussian noise distributions, or an equal variance likelihood for a heterogeneous non-Gaussian noise distribution?\n3. How sensitive are the results to the choice of $\\mathbb P$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for learning DAGs that addresses the challenging problem of uncertainty quantification in causal structure discovery. The main innovation is the development of \"projected distributions\" that have support directly on the space of sparse DAGs by projecting samples from continuous distributions onto the set of acyclic matrices using a combination of acyclicity and sparsity constraints. The authors prove these projected distributions are mathematically valid, develop efficient algorithms for the required projections, and demonstrate through a range of experiments that their method performs well in both accuracy and uncertainty quantification.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a strong paper that makes solid theoretical and practical contributions to Bayesian DAG learning. The core idea of projected distributions addresses an important problem, as most existing Bayesian DAG methods either do not guarantee exact acyclicity or require complex discrete inference procedures. The method appears technically sound and shows improvements over baselines, though gains are often small. The projected distribution idea could likely be applied beyond DAGs to other constrained structure learning problems, increasing the significance of the work.\n\nStrengths:\n\nQuality: The technical execution is very good, with rigorous mathematical foundations and proofs, and comprehensive experimental evaluation in both linear and nonlinear SEMs with multiple metrics and baselines. The variational inference framework is well-designed and the extension to nonlinear DAGs via neural networks is natural, though the method still inherits standard limitations like its cubic complexity and approximate posteriors from variational inference.\n\nClarity: The paper is well-written, with the projected distribution concept presented intuitively and the theoretical framework clearly explained. The novel ideas are integrated nicely with known results.\n\nSignificance: This addresses an important problem in causal inference with practical implications for uncertainty quantification in downstream tasks. The method shows strong uncertainty quantification compared to frequentist approaches and performs well for large sample sizes where other Bayesian methods fall off, with particularly good results on the real protein signaling dataset.\n\nOriginality: The key innovation is to define a novel distribution P(W) over the space of DAG adjacency matrices, which can be used in a variational framework. The approach seems innovative compared to previous approaches, especially this contribution of defining this distribution with continuous support. The entire framework also combines the new ideas with known results/tools nicely. \n\nWeaknesses:\n\nQuality: The method inherits standard limitations including cubic complexity, which limits its scalability (experiments only go to p=100). The use of variational inference does not guarantee good approximation to the true posterior. The non-convexity of the acyclicity constraint means optimization can only reach stationary points rather than global optima.\n\nClarity: While generally well-written, some important experimental details are deferred to appendices rather than being in the main text, and the combination of multiple existing techniques (though novel in combination) makes it hard to work out which components drive the performance improvements.\n\nSignificance: The performance improvements over baselines are consistent but small, and the scalability limitations to networks with around 100 variables may limit practical applicability to larger real-world problems where the method's benefits would be most valuable.\n\nOriginality: While the overall framework is novel, the individual technical components such as the acyclicity constraints, L1-ball projections,  and variational inference) are all existing techniques, so the contribution is primarily in their creative combination rather than fundamental algorithmic innovations. Nonetheless there are some significant innovations in the work.\n\nQUESTIONS:\n1. Can you demonstrate the method on larger networks (p=200-500), or propose any algorithmic improvements that could \nsignificantly increase the scaleability of the approach?\n\n2. Can you determine which components are responsible for the performance improvements? Some ablations could help with this, such as  comparing against a simpler baselines with the same acyclicity constraints without projected distributions. \n\n3. Can you provide recommendations about the settings in which your approach should be chosen over existing methods, given the modest improvements and higher computational cost? \n\n4. Can you provide more analyses as to when the variational approximation works well? This could include diagnostics or an experiment comparing against exact posteriors on a small synthetic problem.\n\nLIMITATIONS:\nThe limitations are discussed above. I do not see any potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilize Bayesian variational inference framework to construct a valid distribution on the space of sparse DAGs for the causal discovery task. The proposed method ProDAG models the prior and variational posterior of potential DAGs. For this purpose the authors use the continuous reformulations of acyclicity constraints and perform gradient descent based on that.\n\nSTRENGTHS AND WEAKNESSES:\nBelow I provide my comments.\n\n# Strengths\nFigure 1 is very helpful to understand the algorithm concepts. The idea of using variational inference for causal discovery appears novel.\n\n# Weaknesses\n\n* The following works also considers a continuous constraint optimization problem\nThe authors should compare their methods to the following works qualitatively and use some of them as benchmarks.\n\n1. Brouillard, Philippe, et al. \"Differentiable causal discovery from interventional data.\" Advances in Neural Information Processing Systems 33 (2020): 21865-21877.\n2. Bhattacharya, Rohit, et al. \"Differentiable causal discovery under unmeasured confounding.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n3. Faria, Gonçalo Rui Alves, Andre Martins, and Mário AT Figueiredo. \"Differentiable causal discovery under latent interventions.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n4. He, Yue, et al. \"Daring: Differentiable causal discovery with residual independence.\" Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021.\n5. Lopez, Romain, et al. \"Large-scale differentiable causal discovery of factor graphs.\" Advances in Neural Information Processing Systems 35 (2022): 19290-19303.\n\n* The authors should provide more details on the “path-following algorithm”.\n* The authors should provide some intuition on “implicit function theorem.” and “Karush-Kuhn- Tucker (KKT) conditions of the projection” in the main paper.\n* The role of $l_1$ ball is not clear. The author should discuss how large or small $l_1$ ball affects the algorithm performance. Also it is not clear how $l_1$ is chosen.\n* It is not clear how the authors obtain $q_{\\theta}(\\tilde{W})$ with variational inference in their algorithm. I would suggest the authors discussing this in detail.\n* The authors should show more experimental results with real-world data. They might check https://www.bnlearn.com/bnrepository/ for this purpose.\n* Figure 2 might be replaced with a table and the space can be used for describing other concepts in detail.\n\nQUESTIONS:\nBelow I provide my questions.\n\n* How do the author set the value for $\\lambda$.\n* The authors said that the projection (2) is unique. Why can there not be two different $\\tilde{W}$ with equal distance from $W$?\n* The authors mentioned that step 1 projects the matrix $\\tilde{W}$ onto the set of acyclic matrices through equation 4. How does the algorithm have access to $W$ to calculate the subtraction here?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ProDAG, a Bayesian structure learning method that employs variational inference to infer an approximate posterior over DAGs given observational data. Unlike prior variational approaches which either (i) enforce acyclicity in the variational family or (ii) modify the prior to penalize cyclicity, ProDAG takes the novel approach of choosing an unconstrained variational family that induces a distribution over DAGs through the use of a many-to-one projection operation. It is shown that this defines a valid distribution over DAGs (weight matrices), and how to propagate gradients through the projection operation. Empirical results show that ProDAG outperforms existing variational Bayesian structure learning methods on standard metrics, particularly as the sample size increases.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this is a high quality paper introducing an interesting approach for Bayesian structure learning. The hard projection step onto weight matrices with a particular sparsity is a novel idea and is shown to lead to strong empirical performance. The idea is backed up with solid theoretical work showing the measurability of the projection mapping and derivation of the ELBO and gradients. The clarity of the paper is also excellent with the key ideas being described well in the main paper with full details presented in the Appendix. \n\nRegarding weaknesses, while the empirical evaluation shows strong performance, I would like to have seen some more insight into *why* ProDAG performs better than existing approaches (see questions below). Additionally, the empirical evaluation compares to variational approaches such as BayesDAG and DiBS that employ augmented spaces. However, for completeness there should also be comparison to other types of Bayesian methods, in particular (i) permutation-based variational methods such as BCDNets; and (ii) MCMC-based approaches such as Gadget [1] which may exhibit qualitatively different behavior (e.g. consistency of MCMC). The related work section should also mention some of these MCMC-based approaches. Finally, a seeming limitation is that ProDAG requires the sparsity hyperparameter $\\lambda$ to be set in advance, and due to the hard projection constraint this may inevitably lead to errors even in the large-data limit (i.e. if one sets $\\lambda < d(d-1)/2$ then one could never learn the complete graph even if it is the true graph). \n\n\n[1] Towards Scalable Bayesian Learning of Causal DAGs. Viinikka et al. (2020). NeurIPS\n\nQUESTIONS:\n- Intuitively, why is it possible to derive the gradients of the optimal solution (Apx C) in closed form but not the optimal solution itself? Given that the iterative procedure only produces an approximation solution to the projection, is there a significant discrepancy between the exact autograd gradients for the approximate solution and the gradients of the optimal solution?\n- Why did the authors choose a multivariate Gaussian variational posterior, given that the true posteriors can be multimodal? (unless the projection step induces multimodality). It would be interesting to see some examples in low dimensions to visualize the effect of the projection step.\n- Did the authors try learning a variational posterior with a Laplace-style prior instead of the hard sparsity projection? If so and it performed worse, are there some qualitative insights into why the hard projection step should be preferred?\n- How did the authors tune the sparsity hyperparameter $\\lambda$ on the validation data? How sensitive is the performance of the method to the choice of $\\lambda$ (i.e. would it be reasonable to simply choose a single $\\lambda$ value and use it across all experiments)?\n\nLIMITATIONS:\nYes, limitations are addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}