Title: Differentiable Structure Learning and Causal Discovery for General Binary Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: differentiable structure learning

causal discovery

discrete data

binary data

DAG
Abstract: Existing methods for differentiable structure learning in discrete data typically assume that the data are generated from specific structural equation models. However, these assumptions may not align with the true data-generating process, which limits the general applicability of such methods. Furthermore, current approaches often ignore the complex dependence structure inherent in discrete data and consider only linear effects. We propose a differentiable structure learning framework that is capable of capturing arbitrary dependencies among discrete variables. We show that although general discrete models are unidentifiable from purely observational data, it is possible to characterize the complete set of compatible parameters and structures. Additionally, we establish identifiability up to the Markov equivalence class (MEC) under mild assumptions. We formulate the learning problem as a single differentiable optimization task in the most general form, thereby avoiding the unrealistic simplifications adopted by previous methods. Empirical results demonstrate that our approach effectively captures complex relationships in discrete data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a general differential structure learning framework and continuous DAG learning algorithm for discrete data that could capture arbitrary dependencies with no distribution restrictions. To address the combinatorial search problems in discrete data, the paper proposes Notears-MLP-Reg method to reduce computational complexity.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well-written with clear structured theoretical investigation and empirical evidence. The novelty is to spot the equivalence of modelling binary data as exponential families with interaction terms as potential candidates for causal parents. \n\nComments: \n- in the paper there are multiple places referring that general discrete models are unidentifiable from purely observational data alone (e.g. in abstract line 8, line 218, line 222) and there are also places restricting to the general multivariate Bernoulli distribution (see line 62, line 245) that indicates single environement /  distribution rather than observational data alone. However, it is well-known that multi-environment observational data alone could identify causal structure [1, 2]. To make the paper more precise, it would be good to clarify the differences. \n\n- Theorem 3 says there exists $\\lambda$, $\\delta$ sufficiently small such that the predicted set is the same as minimal equivalence set. Could the authors discuss in practice (when performing experiments) how to select $\\lambda$, $\\delta$. Is it similar as the theorem suggests, the smaller the better?\n\n- The natural problem in discrete data is combinatorial search. The authors address this problem via proposing NoTEARS-MLP-Reg - which first extracts topological ordering and then fit interaction features consistent with $\\pi$ and restrict to first and second order. Could the authors clarify, under this setup, how did the number of searches scale in terms of number of node increases?\n\n[1] S. Guo, V. Toth, B. Schölkopf, and F. Huszar. Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 36463–36475. Curran Associates, Inc., 2023a\n\n[2] Huang, B., Zhang, K., Zhang, J., Ramsey, J., Sanchez-Romero, R., Glymour, C., & Schölkopf, B. (2020). Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research, 21(89), 1-53.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a general differentiable structure learning framework for causal discovery on binary data from the multivariate Bernoulli distribution, embedding all higher-order interactions without restrictive parametric assumptions. \nIt first provides a non-identifiability result under matching observable distributions. \nThen it shows the proposed algorithm identifies the sparest DAG up to Markov equivalence class under faithfulness. \nExperiments are conducted on simulated synthetic data, compared with a set of causal discovery baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper tackles a broad nonparametric framework that better reflects real-world scenarios, and backs its claims with comprehensive theoretical proofs.  \n2. The narrative is overall well-structured.\n3. On synthetic Erdős–Rényi and scale-free graphs, the method achieves competitive performance compared to existing baselines.  \n\n### Weaknesses\n1. While modelling higher-order interactions is motivated by real-world use cases, a motivational example is missing. Since the experiments are also focusing on synthetic data, it is not clear to me why higher-order interactions should play an important role in real-world causal discovery tasks.  \n2. As noted by the authors, the full higher-order feature map scales exponentially with the number of nodes, limiting its applicability to larger networks. The NOTEARS-MLP-REG heuristic helps in practice but lacks theoretical guarantees, making its connection to the paper’s core identifiability results less clear.  \n3. The notation in Section 4 is difficult to follow. Simplifying or redesigning symbols would greatly improve readability, e.g., by reducing super- and subscripts.\n4. Visualization: Please keep the color-coding aligned for all figures in the paper! For example, FEGS was marked as purple in figure 1 but orange in figure 2.\n\nQUESTIONS:\n1. Following up on the first weakness: it would be helpful to include a real-world motivational example where modeling higher-order interactions is crucial for causal discovery. This would better justify the need for such a general formulation beyond synthetic settings.\n2. What is the computational complexity of the proposed DAGMA-HO method, particularly with respect to the number of nodes and the order of interactions?\n3. Is it always beneficial to model higher-order interaction terms? How should one choose the order of interaction terms in practice? Is there a principled way to select this, or is it treated as a hyperparameter?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses differentiable structure learning for discrete data. Arguing that prior work relies on overly restrictive parametric assumptions, the authors propose a general framework starting from the multivariate Bernoulli (MVB) distribution to capture arbitrary dependencies. This paper proposes finding the sparsest graph, relying on the Sparsest Markov Representation (SMR) assumption to guarantee uniqueness up to the Markov Equivalence Class, and it translates this search into a differentiable optimization problem and introduces a two-stage heuristic to address the primary method's exponential complexity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is exceptionally well-written. The argument flows logically from the limitations of prior work to the general theoretical characterization, the proposed solution, and the empirical validation. The core ideas, definitions, and theorems are presented with precision and clarity.\n\n2. The formulation of the problem as a continuous optimization program is elegant. The connection established in Theorem 3, which guarantees that the solution to the practical optimization problem corresponds to the theoretically desired minimal equivalence class, is a crucial and impressive result that bridges theory and practice.\n\n\nWeaknesses:\n\n1. The paper's solution to the non-identifiability problem relies heavily on invoking the Sparsest Markov Representation (SMR) assumption. Theorem 2, which states that the sparsest graphs belong to a single MEC, is essentially a direct corollary of this assumption rather than a novel derived result. The paper expertly characterizes a problem in Theorem 1, but the principle for solving the resulting identifiability issue is assumed rather than discovered.\n\n2. The experiments are performed only on synthetic data. While this demonstrates that DAGMA-HO works in specific cases where higher-order interactions are known to exist, the lack of any application to real-world data makes it difficult to assess the practical relevance of the problem and the utility of the proposed heuristic.\n\n\nMinor: Format issues -- all appendix references are not rendered.\n\nQUESTIONS:\nI'd appreciate the authors' feedback on the weakness section.\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper studies the problem of causal discovery in a set of discrete binary variables without any specific model. They propose a differentiable learning method that formulates the learning problem as a single differentiable optimization task. They show that general discrete models are unidentifiable from purely observational data and characterise the Markov equivalence class.

STRENGTHS AND WEAKNESSES:
The paper is well-written. The problem is a relevant and important one. 
The logistic formulation of the conditional multivariate Bernoulli probabilities is quite interesting. This helps the authors to form the differentiable form and might be also useful for further researches. 

There are couple of major weaknesses with this work. The contribution is limited given that the previous results on differentiable method for causal discovery such as notears. 

The proposed method also suffers from the limitations of the notears like methods such as getting trapped into local minima, speed of convergence, good initialization, and the choice of parameters such as regularisation coefficient. There are no discussion about these points in the paper. 

Notation B^{j_1,…,j_r} was defined for simpler representation but in some place it has not been used while it should eg. Eq. (3).

References are missing to the Appendix and in some places equations are out of page border eg. (1).

Before remark 2, “The cardinality of E(p) is at most p!”. Why not exactly p! ?
The summation in score function definition should be to n instead of p.
Function (11) is quadratic on [-delta, delta] not only on [0, delta].

QUESTIONS:
Please see above.

LIMITATIONS:
Please see above comments.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a rigorous and well-executed study on differentiable structure learning for general binary data. Its main theoretical contribution is the proof that causal discovery is non-identifiable from purely observational data in the most general binary setting. By leveraging the multivariate Bernoulli distribution, the authors fully characterize the set of graph–parameter pairs consistent with the data, and introduce the notion of a minimal equivalence class to select among these.\n\nThe second major contribution is a novel differentiable optimization framework that recovers the causal structure up to Markov equivalence, without relying on restrictive assumptions such as linearity or additive noise. This formulation enables the recovery of the sparsest DAG consistent with the data, supported by strong theoretical guarantees.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n### Theoretical Rigor:\nThe paper provides a strong theoretical foundation by proving the non-identifiability of causal discovery for general binary data. It rigorously characterizes all graph–parameter pairs compatible with the data using the multivariate Bernoulli distribution.\n\n### Novel Optimization Framework:\nA differentiable formulation is introduced to recover the structure up to the Markov equivalence class, enabling the use of gradient-based methods without restrictive parametric assumptions (e.g., linear or additive noise models).\n\n### General Applicability:\nThe approach handles arbitrary higher-order dependencies in binary data, overcoming limitations of existing methods that assume first-order or linear models.\n\n### Empirical Validation:\nExperiments on synthetic datasets demonstrate the method’s superiority in capturing complex causal structures, especially when higher-order interactions are present.\n\n\n## Weaknesses\n\n### Minor Writing/Typesetting Issues:\nThe paper appears to have been written in a rush. There are several typesetting and referencing errors—for example, broken or missing references to appendices (e.g., “Appendix ??”), which slightly detract from readability and polish.\n\n\n### Limited Empirical Analysis of Optimization Aspects (Minor):\nWhile the authors adopt the DAGMA framework for optimization, the analysis of the DAG constraints themselves is underdeveloped. The paper could be strengthened by a more detailed empirical or theoretical investigation of how the chosen DAG constraint affects optimization performance and recovery quality.\n\nIn particular, recent work suggests that the acyclicity constraint in DAGMA could be replaced with:\n\n- Matrix inversion-based constraints [1,2], if vanishing gradients are a concern.\n\n- NOTEARS-style constraints, if non-convexity of the objective is the dominant challenge.\n\nIncorporating or comparing such alternatives could improve the overall robustness and insight of the paper’s optimization component.\n\n[1] Zhang, Zhen, et al. \"Truncated matrix power iteration for differentiable DAG learning.\" Advances in Neural Information Processing Systems 35 (2022): 18390-18402.\n[2] Zhang, Zhen, et al. \"Analytic DAG Constraints for Differentiable DAG Learning.\" arXiv preprint arXiv:2503.19218 (2025).\n\nQUESTIONS:\nSee weakness parts.\n\nLIMITATIONS:\nMy main concern is about scalability. The size of $\\mathbf{H}$ matrix increase exponentially against the number of nodes in the graph (or at lease cubic if use approximation), which prevents the  algorithm to scale to even graph with near 100 nodes. Also it would be good if the authors can provide runtime comparison of different algorithms. algoriithms", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}